Assistant rewrites links sometimes

Created on 14 January 2025, about 1 month ago

Problem/Motivation

The Assistants API currently is running different plugins - they can be Agents or RAG (or something in-house). So what happens is that the Assistant API gets a question about something for RAG or instruction to do something for Agents and they action plugin then goes to work and finds information or tells the Assistant it is done.

In both those examples the URI's it gives back are deterministic and 100% hallucination free. RAG responses comes from the field in Search API and agents, like the field agent gives back links to field configuration, form configuration etc. of the field it created and edited without using an LLM.

The instruction to the assistant then is to rewrite this according to your site goals, in the case of Drupal CMS it is to find where changes happened on the website so the end user can verify them.

We had an initial issue that was happening a lot (30% of the time), where the link came back relative to the path, instead of relative to the root. This was fixed in 🐛 AI Assistant links are not always relative to root when not absolute Active via code.

However other types of hallucination have been seen in issues:
🐛 Created media fields use autocomplete form widget instead of media library Active
🐛 Links provided by AI assistant are frequently not working Active

Where it has added a hash or an example.com or similar host to the link, causing the link to fail.

Steps to reproduce

Ask it to create fields or something similar for you and after enough times you might get an hallucinated host or other type of rewrite.

Possible Solutions

✨ Add a planning agent Active is one possible solution, but since its still LLM, its not sure it will work.
Having a separate link window, where all the action plugins links are shown, similar to how Gemini references links could be used. Meaning you give each link a number and in the LLM that number references the link without any rewrite.
Doing the same with a rewrite.
Just allowing certain domains to be allowed for absolute link, this will however not work when you use agents that can search and read websites for you.

Remaining tasks

Research which solutions would work the best.

🐛 Bug report

Status

Active

Version

1.0

Component

AI Assistants API

Created by

🇩🇪Germany marcus_johansson

Live updates comments and jobs are added and updated live.

Comments & Activities

Issue created by @marcus_johansson
Comment about 1 month ago →
🇩🇪Germany marcus_johansson
Comment about 1 month ago →
🇩🇪Germany marcus_johansson
Comment about 1 month ago →
🇩🇪Germany marcus_johansson
Comment about 1 month ago →
🇬🇧United Kingdom seogow
LLM will inevitably hallucinate links sometimes. Prompt engineering can get you reliable copy/paste result (on some models, including Gemini/Gemma, Claude 3.5 Sonnet and OpenAI 4o and later), but reliable rewrite is almost impossible to guarantee.

However, because we know the path constraints on Drupal and correct source URLs (in case of the search as a source), we can let LLM just choose as you suggested. That would work without hallucinations.

contrib.social Blog FAQ Discussions

Production build 0.71.5 2024