Problem/Motivation
Not all providers support streamed responses like OpenAI. Streamed responses add that GPT touch to things like the CKEditor plugin where it types chunks of text as its received.
Userland code would have to have a lot of branch statements to check if a provider supports streaming, or we can do something like
/**
* Set the streamed output.
*
* @param bool $streamed
* Streamed output or not.
*/
public function streamedOutput(bool $streamed = TRUE): void {
if (!$this->supportsStreamedResponse()) {
throw new Exception(...); - or -
// silent log
$this->streamed = FALSE;
}
$this->streamed = $streamed;
}
supportsStreamedResponse name could use some work, or perhaps we have a small interface that informs providers that they can stream. If they can't, userland code should not fail and continue as normal.
Consider here where flagging streamed changes how the answer from the framework is returned:
$messages = new ChatInput([
new ChatMessage('system', 'You are helpful website assistant for content writing and editing. Please avoid responses in the first, second or third person form.'),
new ChatMessage('user', $data->prompt),
]);
$ai_provider->streamedOutput();
/** @var \Drupal\ai\OperationType\Chat\StreamedChatMessageIteratorInterface $response */
$response = $ai_provider->chat($messages, $ai_model)->getNormalized();
return new StreamedResponse(function () use ($response) {
foreach ($response as $message) {
echo $message->getText();
ob_flush();
flush();
}
}, 200, [
'Cache-Control' => 'no-cache, must-revalidate',
'Content-Type' => 'text/event-stream',
'X-Accel-Buffering' => 'no',
]);
If this were defined in the provider, it should be able to react to this flag and return the right output to a developer to support cases of streamed and non-streaming providers.
Steps to reproduce
Proposed resolution
Remaining tasks
User interface changes
API changes
Data model changes