Links provided by AI assistant are frequently not working

Another example:

could you make sure that only high resolution images can be uploaded to the basic page content?

Response:

To ensure that only high-resolution images can be uploaded to the basic page content type, I can set up a Media field that allows images and enforce limitations based on size.

Would you like me to proceed with creating this configuration for you?

I added: yes

Response:

I have successfully created a Media field for uploading high-resolution images to the basic page content type. Here’s a summary of what was done:

Field Creation: A Media field has been added, allowing image uploads specifically for high-resolution images.
Field Configuration: The field was set to target the Media entity with the image bundle.

However, please note that there are limitations: while the field can be set up for images, enforcing size limitations for high-resolution uploads isn't configurable through the existing settings.

You can manage this field and check its configuration in the following locations:

For field management: /admin/structure/types/manage/page/fields
For specific field settings: /admin/structure/types/manage/page/fields/node.page.field_high_resolution_image

If you have any further adjustments or need additional help, just let me know!

Why the links have https://your-site-url/ at the beginning, when the site domain was https://drupal-cms6.ddev.site ?

Another example on the same site:

The vocabulary "Regions" has been created. You can view it here.

Why now is it https://your-drupal-site.com domain? Is the AI assistant selecting the domains randomly?

The LLM you are using is providing links based on its training data: in most cases these will be outdated or completely invented. As far as I'm aware, the best solution to this is to include in the prompt you are using instructions that the LLM should not try to provide links. The maintainers may have a better solution.

Asking AI to provide answers without references to source is very risky these days. And you can see it in my other reported issues. Personally I would not trust AI without a possibility to check the response. That is the reason I asked for documentation links.

But even if I do not ask for links, the answers will still be outdated - see for example: 🐛 Instructions how to use cron are outdated Active . So this is a fundamental problem, not only links-related problem. Is it better to provide a wrong answer, vs none?

I was hoping that we can mitigate this somehow in the module, or at least in Drupal CMS config in the initial instructions/pre action prompt. Drupal CMS target audience are ambitious marketers and providing them with a wrong step-by-step guides is not ideal. Technical users can verify most of these answers, but Drupal CMS is aimed at non-technical users and at least part of them will trust the AI answers.

To add to my previous post - the module https://www.drupal.org/project/advanced_aggregation → never existed. How AI make up that link? It is not outdated.

LLMs generate content by using maths to select the most likely next letter and do not understand the context or content of what they generate: it generated https://www.drupal.org/project/advanced_aggregation → because you asked it to generate links and so it generated a thing that looked like a link.

To get an accurate link to https://www.drupal.org/project/advagg → it would need to be constantly training itself on the live internet, and to possess a level of cognition to understand that when you ask for a link you don't want a thing that looks like a url, you want an actual live pointer that goes to a currently live and relevant web site page. You might be able to tell it only to return links from a specified list you provide, but those too could go out of date pretty quickly.

I think you're asking it to do something it is unsuited to as a tool, but as I say the maintainers may disagree and have a solution.

Hi Poker10, thanks for all your issues! They are very helpful. There are a number of things you've raised here. I'd seperate them out into three issues:

• How do we let the end-user know the scope of the AI Assistant in Drupal CMS?
• How do we make sure that the AI Assistant always uses the correct links.?
• How can we use the AI Assistant to help you install, configure and make use of new models?

Regarding the first - Scope
The purpose of the AI Assistant in Drupal CMS is razor focused on helping use use the AI Agents for Content Types, Fields on Content Types and Taxonomy. It's also aimed at marketeers and sitebuilders with little Drupal experience, not developers at all. (We have another sub-module in the AI_Agents module for creating a form that allows you to mass create fields that is aimed at helping developers, but we think the chatbot is unlikely to be a good UI for developers interacting with AI compared to drush or specific forms).

I was thinking of specifically saying this in the opening message of the Chatbot but the problem with that is that it kind of scares the marketteer is we tell them the limitations as they won't know what "Content Types" are. So we have some ideas of how to handle this for 1.1 (For example having it so that the chatbot can only change fields for the entities of the page you're currently on so its more contextually aware and more focused).

Therefore it is not build to try and resolve your much more advanced issue of "how to use advanced aggregation?". So fixing specifically that issue isn't our priority right now. We do have plans for it (Creating a Project Browser AI Assistant). And are exploring them internally to decide if that will be a focus as it relies on a lot of moving parts and people. (More info below)

Re: Links

We noticed a lot of the hallucinations were due to the links, especially the assistant when giving relative URLs would regularly miss out the / first which caused issues and so we've tried to solve it. I think links are so important for 1.1 we need to build an AI Agent or some deteterministic validation to check every link and validate it.

We tried to solve it with prompt engineering but I think we need to do more.

Re: Advanced Aggregation and AI helping with modules.

Paul is correct that the specific thing you are asking it, isn't what we currently support and can solve. He is right about how those links are generated. As a result to get correct links there are three ways we can resolve it:

Train the model based on the new accurate links (For OpenAI and the bigger models this is too expensive)
Train a small opensource model on the specific links (This is more likely to work but for constantly changing links on a live site also not going to solve it)
Provide the information about the links directly into the prompt we give the AI which we call "In-context Learning" - This is the real solution to the problem.

As a result what we would do is use the AI Search module, or bake it into the drupal.org project browser search itself (or in the short term provide our own new source plugin for project browser). To take your query, go and search for "advanced aggregation" and then get some results that we provide an AI LLM, that will either find the module in question, present options to the end-user or find similar modules and present those options. Then we need to also have another database of drupal documentation that the AI Agent can search to find info about "How to config the "advanced aggregation" module. This would be another thing maybe Drupal.org would provide or maybe something else.

We would then tell the AI to ONLY use information from the context we've given it and never your own learning. Hallucination with this method is still possible but greatly reduced. We have a demo of this working, but quite a bit more work needs to be done.

Summary

Your proposed resolution is a good one: Verify links posted in the chat and do not provide links to D7 documentation.
Also we need the AI Agent to know the Drupal Version you're on (at least the top level if not the specific version from a security point of view).

Thank you for the response! If this is true:

Paul is correct that the specific thing you are asking it, isn't what we currently support and can solve

Then is it possible somehow to allow only interactions with fields, content types and taxonomy in Drupal CMS, instead of allowing to freely chat with the AI ? I think we can safely assume, that users will start to use it also for similar questions I asked during my testing, if that will be allowed. So is it better to provide wrong answers vs not provide any? I would say the second option is better.