Contrib.social

Improve handling of RAG/tool output to properly pass as tool messages for better formatting and reasoning

Open on Drupal.org →
Created on 16 May 2025, about 2 months ago

Problem/Motivation

Currently, when the RAG (Retrieval-Augmented Generation) search results are returned from the vector database, the raw tool output is passed directly as assistant messages in plaintext. This causes the AI assistant to treat these results as already “answered” content, preventing it from reformatting, interpreting, or following prompt instructions for rendering recipes, code blocks, images, etc.

The current implementation echoes RAG results directly as assistant message content (plaintext), e.g.:

echo $message->getText();
$full_response .= $message->getText();

This makes the assistant think it has already responded and does not perform further processing or formatting on the results.

Proposed resolution

Instead of passing RAG/tool results as plaintext assistant messages, they should be passed as tool role messages (with an appropriate tool name like search_rag) inside the conversation context. This way, the assistant recognises these as external tool outputs it should reason over, interpret, and incorporate into a well-formatted response.

For example, within the streaming response callback:

  • Detect messages with role tool and add them to the message context with their tool role and name.
  • Do not echo these tool outputs directly.
  • After collecting the tool output, call the assistant again with the full conversation context including the tool messages.
  • Echo the assistant’s final response, which can reformat and clean up the output properly.

This change aligns with best practices in prompt engineering for RAG pipelines, ensuring the assistant distinguishes between raw tool output and its own final response, improving output quality and user experience.

if ($message->getRole() === 'tool') {
  $this->aiAssistantRunner->addToolMessage('search_rag', $message->getText());
} else {
  echo $message->getText();
  $full_response .= $message->getText();
  ob_flush();
  flush();
}

Then request the assistant’s final formatted response and echo it.

📌 Task
Status

Active

Version

1.0

Component

AI Search

Created by

🇪🇸Spain gxleano Cáceres

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

contrib.social Blog FAQ Discussions
Production build 0.71.5 2024