Problem/Motivation
Currently we only allow you to return the following data in the streaming interface:
This is subject to change with
📌
Add token usage to streamed chat
Active
where token usage might be normalized as well.
We also need to normalize the tools usage in the StreamedChatMessageInterface that can take the tools response and put it in a getTools and setTools method. Since the tool might not be formed yet and it makes no sense to call the tool or the normalizer before that and it should not be normalized at that stage.
So what we should do is on the StreamedChatMessageIterator that is the abstract class, we should add a variable on the createStreamedChatMessage that adds the mixed $tool part and then adds this to a tool class.
On the StreamedChatMessageIteratorInterface we should force a method called normalizeTool, where each provider has to have a method for normalizing that tool. This can be added to the abstract method, to not break providers and be forced in 2.0.0
This should also be sent to the PostStreamingResponseEvent so anyone that needs to get the tools after it streamed out the text, can get it.
This is the steps of a streamed tools call, just for reference for a mental model:
data: {
"choices": [
{
"delta": { "role": "assistant" },
"index": 0
}
]
}
data: {
"choices": [
{
"delta": { "content": "Sure! Let me check..." },
"index": 0
}
]
}
data: {
"choices": [
{
"delta": {
"tool_calls": [
{
"id": "tool_abc",
"type": "function",
"function": {
"name": "get_weather",
"arguments": "{"
}
}
]
},
"index": 0
}
]
}
data: {
"choices": [
{
"delta": {
"tool_calls": [
{
"function": {
"arguments": "\"location\":\"Berlin"
}
}
]
},
"index": 0
}
]
}
data: {
"choices": [
{
"delta": {
"tool_calls": [
{
"function": {
"arguments": "\""
}
}
]
},
"index": 0
}
]
}
data: {
"choices": [
{
"delta": {},
"finish_reason": "tool_calls",
"index": 0
}
]
}
data: [DONE]
Steps to reproduce
Proposed resolution
In StreamedChatMessageInterface/StreamedChatMessage add getTools and setTools and add it to the constructor.
In StreamedChatMessageIterator add a variable called tools
In StreamedChatMessageIterator add tools to the createStreamedChatMessage method and add those to the tools
In StreamedChatMessageIteratorInterface add a method that is normalizeTools that each provider has to take care of to normalize the tool output according to how its already done.
In StreamedChatMessageIterator add a default method that does nothing, just to not have breaking changes. This will be removed in 2.0.0.
Add the tools to the PostStreamingResponseEvent, via the constructor.
In the PostStreamingResponseEvent add a getTools and setTools to add the tools.