Move the results to the system prompt instead of Assistant Message

Created on 15 January 2025, 7 days ago

Problem/Motivation

Currently the Assistant will have a system prompt, that comprises of the Assistants System Prompt + Pre-Action Prompt + Instructions. The Pre-Action Prompt is only rendered when there is not action taken or any results given.

If the assistant uses one of the actions, that action will return the answer in an assistant message currently.

The current Drupal CMS instructions wants there to be a two step process, where you approve something first and then apply it via the actions (agents in this case). The problem is, as can be seen in the issue queue, that it can hallucinate what it has done.

The working theory is that happens because of the message history and the LLM, messing up what the results are.

The idea here is to move the results to the system prompt instead and specifically write "No actions have been taken" if there has been no actions taken, to enforce to the LLM that its not confusing.

Proposed resolution

Move the results from the assistant message to system prompt.
Test it against the previous solution manually.

πŸ“Œ Task
Status

Active

Version

1.0

Component

AI Assistants API

Created by

πŸ‡©πŸ‡ͺGermany marcus_johansson

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @marcus_johansson
  • Merge request !408Resolve #3499935 "Move the results" β†’ (Merged) created by marcus_johansson
  • πŸ‡©πŸ‡ͺGermany marcus_johansson

    So the needs review here, is rather that if you run Drupal CMS, to test it with this MR instead of the AI module in the recipe. From my testing (~50 messages) it looks way more consistent.

  • πŸ‡¬πŸ‡§United Kingdom MrDaleSmith

    Looks good to me, and seems like a useful thing to get in ASAP.

  • πŸ‡ΊπŸ‡ΈUnited States zengenuity

    After applying this patch, it does seem to be somewhat better. If I ask it to "copy all the fields and their configuration from the Case Study content type to the Project content type," it sometimes responds that it can't do it or didn't do it, instead of previously always saying it could and did do it and failing. (Case Study in this case is from the Drupal CMS recipe, and Project is a type with no fields.) I have still found some instances of it saying it worked, though, even though it didn't. Not totally consistent, but I think it's better than before the patch.

    I'm still getting one error consistently, though, which is perhaps not what this issue is meant to address. If you ask the bot to copy a specific field from a content type, it will make a new one that is often not the same type as the original. The example I've been using is the Client Logo field in Case Study. It's a media reference field. Consistently, the bot correctly plans to make a new field with the same type and configuration, but then when you okay that plan, it makes a new image field instead. It then reports that it copied the field over, using the same form and display settings, which can't possibly be true, given it's the wrong field type. In this case, it has taken an action, but it misreports what it did. That may be a separate issue, but I just thought I would mention it.

    In any case, in my testing, this patch has improved the bot's answers in some instances, and doesn't appear to be making it worse.

  • πŸ‡¬πŸ‡§United Kingdom MrDaleSmith

    I think that issue would be worthy of its own support issue - at least partly so we could get the improvement in this patch in as quick as possible. Would you be able to raise a new ticket for that? It sounds like something we should at least be able to replicate.

  • πŸ‡©πŸ‡ͺGermany marcus_johansson

    Thank you Paul for the new issue and thank you zengenuity for testing it. I have done further testing and I am also seeing minor improvements, so this is getting merged.

Production build 0.71.5 2024