Contrib.social

Improve the StreamedChatMessageIterator without breaking changes

Open on Drupal.org →
Created on 28 July 2025, about 1 month ago

Problem/Motivation

Currently the StreamedChatMessageIterator is suboptimal - it doesn't expose all the metadata that the normal chatoutput offers, it also has its own structure when the outputted data in the end is usually comparable. The events are not trigger reliabily and the generator is not extendable at the moment.

The issue 📌 Add token usage to streamed chat Active have already improved token handling.

We have both ✨ Allow tool calling in streamed chat Active and 📌 Support Fibers for collaborative multitasking on LLM io waiting Active that basically needs the same thing. A way to collect all the streaming data and create a ChatOutput/ChatMessage from it. We should solve this independently

Proposed resolution

  • Add the interface doIterate where the provider have to yield the method createStreamedChatMessage.
  • Move it so that the default getIterator exists in the base class, where it is triggering the doIterate.
  • Add so you can add tools to the createStreamedChatMessage.
  • Add the method populateChatMessage that each provider should extend. Add an open ai based version in the base class, so its not breaking.
  • Add the method populateTools that each provider should extend. Add an open ai based version in the base class, so its not breaking.
  • In getIterator, run the assembly of the tools and the messages and then trigger the event.
  • Rework the trigger event method, so there are no doublets in there.
  • Write tests for this.
  • Document it.

API changes

This does change a lot how the stream iterator works, however it will be ok since:

1. Any OpenAI based provider will just work from 1.2.0
2. AWS Bedrock we can control, only Gemini will need rework and is baed on external persons work. We can however provide a path.
3. It is backwards compatible in that, if nothing changes, it still work 100% with streaming since they implement the method that the end-user is using. It will just not work with streaming and tools and the events.

✨ Feature request
Status

Active

Version

1.2

Component

AI Core module

Created by

🇩🇪Germany marcus_johansson

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

contrib.social Blog FAQ Discussions
Production build 0.71.5 2024