Allow tool calling in streamed chat

Created on 5 June 2025, about 2 months ago

Problem/Motivation

Currently we only allow you to return the following data in the streaming interface:

  • Text
  • Role
  • Metadata

This is subject to change with 📌 Add token usage to streamed chat Active where token usage might be normalized as well.

We also need to normalize the tools usage in the StreamedChatMessageInterface that can take the tools response and put it in a getTools and setTools method. Since the tool might not be formed yet and it makes no sense to call the tool or the normalizer before that and it should not be normalized at that stage.

So what we should do is on the StreamedChatMessageIterator that is the abstract class, we should add a variable on the createStreamedChatMessage that adds the mixed $tool part and then adds this to a tool class.

On the StreamedChatMessageIteratorInterface we should force a method called normalizeTool, where each provider has to have a method for normalizing that tool. This can be added to the abstract method, to not break providers and be forced in 2.0.0

This should also be sent to the PostStreamingResponseEvent so anyone that needs to get the tools after it streamed out the text, can get it.

This is the steps of a streamed tools call, just for reference for a mental model:

data: {
  "choices": [
    {
      "delta": { "role": "assistant" },
      "index": 0
    }
  ]
}

data: {
  "choices": [
    {
      "delta": { "content": "Sure! Let me check..." },
      "index": 0
    }
  ]
}

data: {
  "choices": [
    {
      "delta": {
        "tool_calls": [
          {
            "id": "tool_abc",
            "type": "function",
            "function": {
              "name": "get_weather",
              "arguments": "{"
            }
          }
        ]
      },
      "index": 0
    }
  ]
}

data: {
  "choices": [
    {
      "delta": {
        "tool_calls": [
          {
            "function": {
              "arguments": "\"location\":\"Berlin"
            }
          }
        ]
      },
      "index": 0
    }
  ]
}

data: {
  "choices": [
    {
      "delta": {
        "tool_calls": [
          {
            "function": {
              "arguments": "\""
            }
          }
        ]
      },
      "index": 0
    }
  ]
}

data: {
  "choices": [
    {
      "delta": {},
      "finish_reason": "tool_calls",
      "index": 0
    }
  ]
}

data: [DONE]

Steps to reproduce

Proposed resolution

In StreamedChatMessageInterface/StreamedChatMessage add getTools and setTools and add it to the constructor.
In StreamedChatMessageIterator add a variable called tools
In StreamedChatMessageIterator add tools to the createStreamedChatMessage method and add those to the tools
In StreamedChatMessageIteratorInterface add a method that is normalizeTools that each provider has to take care of to normalize the tool output according to how its already done.
In StreamedChatMessageIterator add a default method that does nothing, just to not have breaking changes. This will be removed in 2.0.0.
Add the tools to the PostStreamingResponseEvent, via the constructor.
In the PostStreamingResponseEvent add a getTools and setTools to add the tools.

Feature request
Status

Active

Version

1.2

Component

AI Core module

Created by

🇩🇪Germany marcus_johansson

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Production build 0.71.5 2024