Assistant not giving proper response

I am on the dev version with commit ID "591c0ee59e50619861cb81fe39ddbb84a0d25dc9".
The issue still exists.
I am getting a mix of JSON and text explanation.
I asked ChatGPT and it gave me the response: https://www.drupal.org/files/issues/2025-01-03/model_response.md →

So the idea that I liked here is that we can tell it to always respond in JSON.
If some action is there, it will respond with the action key, if no action, then it will respond with some other key, eg. {text: ''}.

Then in our code we can check if text or action and proceed accordingly.
cc: @marcus_johansson

We're probably going to need details of your set up to try and recreate this - the prompts used for the assistant and the set up of the VDB provider, and also your SAPI back-end. It would also be useful to have the questions you asked and the responses you expected.

@vivek - is this still an issue. We could chat on Slack if you need help with a setup. Otherwise I would close this?

I haven't used that chatbot in a while.
Will need to update the codebase and test it again.
For now I am closing this.
I'll re-open this if I face the same issue again.

I can reproduce this issue in 1.1.0-beta1. It appears that when the allow_history option is set to either session or session_on_thread, the Rag Tool in this case—is not being properly initialized.

As a result, we're seeing responses similar to those described in this issue.

Everything works fine if we are not allowing history.

@gxleano, @vivek panicker I have faced this issue previously and when I changed the prompt with an additional instruction like "You are a looping agent that can first search and try to find information and then answer.
When you use the RAG tool, make sure that you only search for 5 items, unless the query is very complex. If you need to do multiple searches, you can trigger them all in the same go."

I was getting proper results. Can you try this from your end?

Hey @anjaliprasannan

Thanks for your feedback. Unfortunately, it didn’t work on my end. Honestly, I don't think this should be handled through the prompt, this seems like a feature that should work out of the box as part of the tool.

In order to have more context about the issue, this is the stack that I am using to reproduce it:
- Drupal 11.1.7 with Umami

Modules installed:
- ai
- ai_api_explorer
- ai_assistant_api
- ai_chatbot
- ai_search
- key
- search_api
- ai_agents
- ai_provider_openai
- ai_vdb_provider_milvus
- ai_agents

I will set this to release blocker, until we have this figured out. It might be that the actual solution is in AI Agents module though.

You are using 1.1.x for all the AI modules above and set it up freshly, correct?

You are using 1.1.x for all the AI modules above and set it up freshly, correct?

Yes, all the modules in the AI stack are coming from 1.1.x

I will try to spend some time today in order to debug it and can add more information about the issue itself.

Here we have an example of wrong output:

Frankly, if this is the only issue blocking the release, happens only under certain conditions and only with VDB/RAG enabled, I would move it to 1.2.x and have the release without the fix.

@valthebald - There is one more that is kind of done, but no the results of it - we have built a whole agent test framework here: 📌 Create visible testing framework Active and I'm currently writing tests for the common things you do in Drupal CMS. When we can get that to pass to 90% on any run, 1.1.x is ready.

However this tool would most likely help to figure out this issue as well, since it will be able to set any assistant/agent in a specific history state an rerun over and over.

Even I am getting this error now after updating my site to latest version of AI (1.2.x-dev) and AI Agents (1.1.x-dev) modules. PFA the screenshots.

@vivek panicker - could you confirm that the provider module is also updated to 1.1.x?

@marcus_johansson Yes, the ai_provider_openai is also updated to 1.1.x-dev version.

I probably found the issue while working on this 🐛 Only show agent assistant form, if AI Agents 1.1.x+ is available Active . The agent that the assistant is connected to, did not save correctly. As soon as that is merged, please retry to create a new assistant with rag and see if it works.

I've been testing this issue on 1.1.x-dev and everything works fine now when Allow history option is enabled.

See evidences:

Moving this issue to fixed as it has been already released into 1.1.0-rc1

Automatically closed - issue fixed for 2 weeks with no activity.