Recent comments

🇬🇧United Kingdom yautja_cetanu

Prioritization for AI Jul 14th 2025

Bold means required for next imminent release. Italic means Marcus will work on it this week.

Also note that any issue for Experience Builder Agents we are reactive on - I do it directly when they need it.

Core

Field Widget Action

AI Automators

AI Agents

  • Create the concept of Guardrail agents Active - Guardrails. I think we can start with the backend logic for it and add the UX/UI later. It will be configuration entities, with a plugin system, that much is known. It’s also one of those issue where development means discovery.
  • Add artifacts to agents Active - Artifacts. We are doing more and more long research and web scraping when 1.2.0 is released. Artifacts are needed.

MCP Client

  • 📌 Write down issues here Active - I have to write down all the issues needed there. Rob Loach is working on it, so I don’t want to block him.

Needs Review (from Marcus)

Needs Review (urgent)

🇬🇧United Kingdom yautja_cetanu

We currently have a tool in the AI agents module that can list fields. So we want to?

  • Either call the Field Agent itself.
  • Or call the field tool itself
  • If we are writing content to the tool we probably need the content seeder and base tools with permissions to edit that entity.
  • We could duplicate the field tool or extend it, or put parameters on the field tool so there are ways of filtering fields without hard coding a list.
🇬🇧United Kingdom yautja_cetanu

Follow up

  • Need to turn it into a proper agent now file upload hasn't happened.
  • Make it so we can swap out the back-end service for looking at the images.
  • Make it so we can configure the number of loops
🇬🇧United Kingdom yautja_cetanu

Before doing this we need to decide:

  • What service are we using to help AI understand attachments? (Unstructured.io for example?)
  • Or are we using a service that takes a file and generates a screenshot and then passes that?
  • Once we have decided we will want a new operation type and abstraction layer to make it so different services can be used and make sure it degrades gracefully if these services are not available.
🇬🇧United Kingdom yautja_cetanu

Random thoughts:

- How do they handle platform with all the metadata about providers and models?
- Do they have some concept of "Operation Types", some way of mentioning the skills of
- How do they handle multiple providers and models coming together with a consistent approach to logging (Such as cost metadata?)
- How do they handle tool logging, and tools within the code not just external tools..
- Do they have some consistent manner of handling permissions in the tools?
- The Agents, what theoretical approach to agents do they take (Swarm? Orchestrator? All?).
- Do they have my concept of agents creating a "workflow" so that php handles the orchestration of tasks not AI handling the orchestration of tasks. For example "go through these 100 piece of content and check for PII"

(Reading through their docs and answering it)

🇬🇧United Kingdom yautja_cetanu

"it can immediately raise an error, which stops the expensive model from running and saves you time/money."

From the OpenAI Agents SDK, I can't see if Guardrails HAVE to stop the execution or CAN cut the operation. I think it should be the "default" approach for guardrails but not the only. Instead it should be possible (even if not possible with version 1.1 to)

  • Have guardrails stop the execution.
  • Have guardrails go back to the agent to give them another go at it.
  • Have guardrails trigger some kind of end-user action that would allow the execution to continue where its left off. Terminate it, or start down a new path.

I don't think we should build all the above options above. But I think we should think about it as a possibility.

🇬🇧United Kingdom yautja_cetanu

Summary from the call:

  • Plans to put it in the top right in Gin, had hit roadblocks as the top right is for saving in Gin.
  • Experience builder puts it in the top left do we want it there?
  • However the XB approach has a top bar that goes across the full width. In Gin the top bar stops before the left side bar so its not obvious we can put it in the same place. There are a number of options below to explore.
  • Showed the new full height chatbot.
  • We said that we want to take an AI first approach and make the AI prominent, therefore we are ok with suggesting changes to the approach Gin has rather than assume AI has to fit within existing design approaches.
  • The approach XB took with its designs "What can the AI do? and how can we convey what AI can do?"
  • Renee would share the benchmarking
  • We think the overlay dimming the page is unlikely to be what we want as we think the chatbot will likely be linked to context and information on the page.
  • When it opens, isntead of overlaying, can we make it so it pushes the page accross.

Options:

  • Do we put it in the top right anyway (next to the sidebar)?
  • Do we go for the left left to the right of the drupal logo? (In the Gin toolbar or the top level toolbar?)
  • Do we make it so its below the drupal logo (Alongisde search, Dashboard)?
  • We also explored the bottom right (Renee, thinks that this meant some users thought it was a support thing not AI).
  • There is some exploration of whether or not its a draggable model that you can move around.
🇬🇧United Kingdom yautja_cetanu

Descisions from a call on 10th July:

  • Bucket 1: We aim to completely replace the existing AI logging module.
    • The AI logging module will require https://www.drupal.org/project/extended_logger
    • We will move to the format of logs being stored in the extended logger entity where most of the data is in the custom Data field
    • We need an upgrade path + Migration of current entities to the new entities (Or introduce a new module (AI logging Deprecated)
  • Bucket 2: We make some simple views in the AI logging module to do things like monitoring costs, etc
  • Bucket 3: We make a method of exporting logs (to a file system or database system
  • Bucket 4: We would create another module for bringing the AI logging into OT, to do more advanced observability. This would maybe include work with Grefana
    • Receive work and convert it into traces and spans
    • What to send and how to send? (They think this won't be the same as in the simple version?) (Alexy thinks that the events will be more developed events and just logs. (Jamie thinks that this will be the same as most advanced stuff needs to go into the DB version)

Things we will need to think about:

  • Abstraction layer, if different API's report back token cost differently, we need to handle bringing it all into one place
  • We will likely want AI to summarise the logs (similar to the OpenAI logging module we had)
  • We might want to look at the OpenAI SDK to see if we can create custom attributes to allow us to do similar things to what OpenAI Agents SDK does.
  • If we store all the information in the AI logging, we may need to provide the ability for provider modules to tell us what syntax they use for similar metadata (like Request cost)
  • We need to think about how we bring Search AI results and tools alongside this
  • Show Alexy the UI we used for LLM logging right now
  • We probably want the AI to summarise each span to increase readability
  • Do we want to make use of Michal's UI for LLM logging of agents
  • Do we have some way of categorising AI logs depending on purpose and doing different things with metadata (we log agent calls, differently to AI translate, or simple API explorer)
🇬🇧United Kingdom yautja_cetanu

So the questions I have abotu this as someone who understands OT if you could help me murz?

- My understanding is OT is primarily useful when we want to aggregate logging metadata across an organisation and multiple different use-cases software. If all you wanted was to know how much money you've spent with Claude for your single drupal site, OT integration would be overkill right?

- There is an issue that the OpenAI Agents SDK solves which uses an architecture of traces, group ids (for chat history) and spans to show you a trace of the agent workflow across LLM calls and tool/ function calls which helps you understand WHY the agent architecture ended that way.
- Similarly you might want to log a trace of an ECA workflow where each span is the output of a specific piece of logic inside a node on the graph.

My understanding is that OT is not an out of the box solution to either of these problems? it's not primarily for logging and debugging a workflow of functions inside code, but for monitoring api calls and information transferring between systems. My understanding is you COULD use OT for this if you wanted to, creating your own approach to traces and spans for agent work flows and make them fully compatible with OT, but OT doesn't necessarily give you anything towards that goal?

Is this correct or have I missunderstood OT?

🇬🇧United Kingdom yautja_cetanu

Great! Couple of comments

- This is not exactly the same as the XB AI design. I think that's fine but maybe we should check with Lauri before we merge (personally I think it's likely to be fine so we should go ahead and get working on it right away
- there are new features the xb AI adds (Details, list of tasks it needs to complete, suggested prompts, upload of files). But I think this should be a follow up.
- the biggest issue is the placement of the AI chatbot button and it's icon differs in xb AI. We probably need some user testing or something on that (maybe follow up) to see if people can find it. The current approach to the chatbot is very easy to find although because it's always there it's also a little annoying

- Do we think the chatbot should go over the page content or push it to one side?
- How much would that chatbot have its own style and how much would it inherent from the site admin gin theme?
- we definitely need a setting on each chatbot for different styles as the back end chatbot and front end chat bot will likely want to differ. I think we already have that though but if we go the full height I think we definitely need that.

🇬🇧United Kingdom yautja_cetanu

One thing about this, it's likely that more and more modules will have some AI functionality alongside the other categories.

I wonder if AI utilities Vs AI enabled or something should exist?

- the AI vector database provider is one tool.
- but a webform agent that uses AI to create forms should maybe be distinct ?

It's kind of fine if modules can have multiple categories though

🇬🇧United Kingdom yautja_cetanu

From a look and feel this is just better than what exists now however:

- It feels mostly a repaint of what exists now. One feedback I got from what exists now is that things like "Modification tools" - Sub-agents tools were confusing to people.
- We have the ability to configure the tools but it appears at the bottom of the page. Which is confusing.
- The UI for configuring the tools is really bad (just having to know the machine names of all bundles for example, know that you would have to limit the content seeder by entity type etc).
- Cannot have multiple versions of the same tool.

However, as mentioned the would get solved with Base Tools.

So given the above is just better in every way than what we have now, happy for this to go in. But it will likely need to be rethought from scratch with base tools

🇬🇧United Kingdom yautja_cetanu

Yes that may help, the issue is I think we need this Base Tools things figured out. So for me, I think having some Designs of how the Base Tools, Tools API will work is important first and then we want a plugin finder/ browser imo. Base tools may not change the browser that much but I think it might. I also think we want to use AI to help people find the right tool.

Also can we see some designs and screenshots of it here?

🇬🇧United Kingdom yautja_cetanu

"Go to the configuration page at /admin/config/media/advanced-aggregation."

This bit of it, we haven't fixed yet right? I've seen the links still regularly get output incorrectly.

🇬🇧United Kingdom yautja_cetanu

I think the important thing with entities is that it should be very obvious when you create a new content type and bundle that you can make a tool for it with a permission. Something like

- Create an AI Tool for Agents to interact with this entity.

🇬🇧United Kingdom yautja_cetanu

@Catch, Should we add something explicit then:

"I will cooperate with the Drupal Security Team as needed"

  • By Opting into the Security Team you and your maintainers are taking accountability of the code submitted whether it is from third parties in the issues queues or created with AI. All code submitted under security review requires human oversight and you can confirm that you, or another human has been through the code and considered the security implications.
  • Where AI has assisted the creation of the code you are explicit on the degree to which AI has written the code and a general process of how you have reviewed or worked with AI

I don't know how easy it will be to enforce. But I do wonder if, its ok to submit a MR, purely vibe coded, but we have something that clearly says it has somewhere. But before its commtted to a branch that will be under securtiy reveal and released, we have a policy that people need to have had human review of the code?

🇬🇧United Kingdom yautja_cetanu

Yay! The thing I vibe coded helped maybe!

🇬🇧United Kingdom yautja_cetanu

Sorry I mean can we post screenshots of any changes to the UI. (I think its the other issue though.

Ok happy to go with what you've suggested:

- Can we confirm the bubble only pops up when its placed explicitly in the backend? (If we want it to appear by default I think maybe we could put it as a default on setting in the general toolbar).
- Ok makes sense. Looking online it seems like "Bubble" is not commonly used outside of CKEditor, but this is a CKEditor Admin screen and as you said, contextual suggests other things that this is not.

🇬🇧United Kingdom yautja_cetanu

ah ok! I thought this was what the UI would look like to the end-user.

Could we get a screenshot of what the UI looks like now with this change? Could we get something that explains the difference between "Balloon Menu"

Should it becalled "In-Context Menu" or something? That suggests it is what you get when you right click? Is "Balloon Menu" a commonly understood term?

It seems like there were two issues:

- Reporter felt they didn't understand why these two exist and what the purpose of the second one was.
- There was a bug that made the graphics look off if you had both.

I feel like the second has been fixed but do we think the first has been fixed? (If we could get a screenshot, it would be good to know if this makes sense to sirclickalot)

🇬🇧United Kingdom yautja_cetanu

What is the purpose of the baloon menu CKEditor Toolbar Button itself? Does it do anything?

🇬🇧United Kingdom yautja_cetanu

@Artem - What is the purpose of the Baloon Menu appearing in the Toolbar? Does it do anything if its there? Could it instead be a setting inside the CKEditor Assistant Toolbar?

🇬🇧United Kingdom yautja_cetanu

It seems there are two things going on here:

- One is there is a very real question that needs to be addressed quickly of how to handle vibe coded modules of poor quality.
- Then there is a question of handling this particular individual.

Re: the first issue:

Obviously I am a strong user and advocate for AI and AI in coding. However, I've seen first hand the kinds of mistakes AI can make that are quite difficult to detect in some automated way. (Necessarily because if it is, AI is quite good at reviewing them quickly and fixing it). Security issues and architectural issues of code not using Drupal correctly are going to become an issue. If we allow this to go unchecked without clear rules the sheer volume may become difficult to handle.

It does seem there should be policies made and this can be a case study on what the policy should be for example:

  • We should probably require people who write code or commits or patches to say if they have use AI to create it, how they have used AI and how much. This will be difficult to enforce but then it would justify "going through slack messages"
  • We should have some policy for how we go about going from Beta to Release and the level of care we expect from people. That is perhaps stronger than before given how easy it is to vibe code something.
  • There has been clear cut evidence of people using AI to write comments on issues to gain credit and this should be addressed. Instead of spam there could be something to do with "Low effort" comments
  • There should probably be some way of warning individuals and maybe companies that engage in this behaviour rather than immediately punishing people

Re: the way this thing is handled.

Struggling to know how to reply to this in light of Newtoid's comment which I fully agree with. There is something about the tone of many comments that seem targetted and personal.

I think its because behind the comments that seem "mean", there are real actionable important things that I think are important to turn into rules that people can follows and actions people can take to avoid punishment.

I think I would second Darren Oh's list but have something where we make it explicit a level of human effort expected from people (To address dww's comment). It seems like we need some level of expectation when someone has privalledges that their own human efforts are judged and held accountable.

Disclaimer: The person in question reached out to me and asked for my comment. I don't know them, nor have I looked at the modules specifically. I don't have a strong view of whether or not they should lose their vetting status but just think that the issue is important that the reasons why and how to avoid it (including this person) should be clear.

🇬🇧United Kingdom yautja_cetanu

The actual problem is that the provider is missing the feature to provide "tool responses in the streaming response".

This will require looking at the OpenAI SDK and its function calling features and making the provider make use of the function calling streaming features.

🇬🇧United Kingdom yautja_cetanu

Excited that you're getting into it! I just had a call with the eca team and will be uploading it to YouTube and also we've shown our ai agents working with the action plugins at drupal camp England this weekend. It's all in a branch of the AI module 1.1.0

🇬🇧United Kingdom yautja_cetanu

I don't think so... I think two enhancements to this module:

- Have the alt text generation understand the text it appears it and the content.
- Have it be passed the language.

This are small changes.

This feature is awesome btw!

🇬🇧United Kingdom yautja_cetanu

We've been exploring an automated test framework here: https://www.drupal.org/project/ai_test_ui
This tests just the agents bit of it and then we'd need another thing for testing Assistant prompts with chat history connected to the agents.

🇬🇧United Kingdom yautja_cetanu

Can we add (There are loads of warnings for this but its caused so many issues)

Note: You will need to be on a paid Tier for the AI features to work.

I agree that this isn't an ideal solution but it's better than it currently is until we figure out longer term solutions (in browser models, some default that doesn't need an API key)

🇬🇧United Kingdom yautja_cetanu

Could you run a test of how well Gemini works with the Drupal CMS Assistant.

https://docs.google.com/spreadsheets/d/1zzqJTxCboYPTnpA33Kt1cDg7gQzPh2g3...

Could take a copy of this spreadsheet and run it manually for Gemini. I still don't know the process for deciding which providers are included.

🇬🇧United Kingdom yautja_cetanu

Note: If we try and align with MCP below is the schema MCP uses for tools. (This is probably for a new issue)

  • I am not suggesting we use ALL of MCP for tools. Just the schema for how an LLM picks between tools.
  • Probably not ALL tools will be ECA Actions, but all ECA Actions can be tools. (For example "Handover between agents tools"). We would maybe make a parent of ECA Actions called AI Actions?)
  • I think it might be good to put this as a layer between LLMs and maybe ECA Actions.
  • We need to think about how we allow a site owner to insert their own config into the action. So there is a provided "Description" by the module but a site owner can write their own copy and add it to the description for their own prompt engineering
        "Tool": {
            "description": "Definition for a tool the client can call.",
            "properties": {
                "description": {
                    "description": "A human-readable description of the tool.",
                    "type": "string"
                },
                "inputSchema": {
                    "description": "A JSON Schema object defining the expected parameters for the tool.",
                    "properties": {
                        "properties": {
                            "additionalProperties": {
                                "additionalProperties": true,
                                "properties": {},
                                "type": "object"
                            },
                            "type": "object"
                        },
                        "required": {
                            "items": {
                                "type": "string"
                            },
                            "type": "array"
                        },
                        "type": {
                            "const": "object",
                            "type": "string"
                        }
                    },
                    "required": [
                        "type"
                    ],
                    "type": "object"
                },
                "name": {
                    "description": "The name of the tool.",
                    "type": "string"
                }
            },
            "required": [
                "inputSchema",
                "name"
            ],
            "type": "object"
        },
🇬🇧United Kingdom yautja_cetanu

Definitely todo:

Fallback for no LLM summaries.

- Requested Actions - Goes to> "First few lines of last user prompt"
- Performaed action- Goes to> "First few lines of that is the body of the response"

Follow up issues:

Explore how good smaller cheaper models are
Explore embedding a small opensource llm to handle this.

🇬🇧United Kingdom yautja_cetanu

As mrdalesmith mentioned in the other comment:

"Just to note, the AI Key is checked as it is saved against the provider settings form for OpenAI so this particular error would be best handled by the providers (as each provider may need a different process to check if the key is usable).

Do all providers return these errors in a normalised way we can check for consistently?

"

🇬🇧United Kingdom yautja_cetanu

On AiQuotaException write a message that there are techincal issues and talk with an administrator.
On AiSetupFailureException write a message that there are techincal issues and talk with an administrator (this should not be able to happen really).

What are these? Is QuotaException running out of credits? If so

"I am sorry, something went wrong. This could be due to the attached account requiring more credits."

🇬🇧United Kingdom yautja_cetanu

Ok firstly this is amazingly cool!!!
So the good things we need to do are:

  • Maybe create a video explaining it.
  • Maybe a blog post showing screenshots of each page and why its useful and how to use it.
  • Would love to bring emma into it and see what she thinks
  • Tried downloading a prompt and used it in playground and it gave roughly the same answer as in drupal. Super cool for figuring out what is going on.
  • Connect this with Evaluations.
  • Have a button on DeepChat (There needs to be a setting to turn this on) that allows you to highlight an assistant message and go to the log directly.

Couple of points:

  • What happens if people don't want to spend money on API calls to summarise any log. Is there a setting to turn this off and is there a fall back?
  • It would be good to be able to export the log itself and see a similarly well structured log file that is human readable but also machine readable. I'd like it so that multiple front-ends could visualise the logs differently. (For example a react tree of agents, minikanban could replay the logs)
  • I'm personally finding myself lost with the "Child" logs and where the one I'm looking at fits in and relates to the child logs. Don't have an immediate solution for it yet
🇬🇧United Kingdom yautja_cetanu

Given how much energy there is behind this could you write in the issue what schema you're aiming for?

It woild be nice to see how it differs from the Mcp tool schema or pattern and why?

🇬🇧United Kingdom yautja_cetanu

Just for our info.

Did upgrading Acquia DAM and upgrading it to Drupal 11 fix it? OR did you remove Acquia DAM to fix it?

🇬🇧United Kingdom yautja_cetanu

From the sounds of things that operation end type looks like it woild be good in the module? An embedding with llm endpoint?

Similarly the reranking endpoint seems like a good one for the ai module.

Maybe it's worth including though anyway bevause would it be that bad if the future has end points being in their own api modules?

Like you could have a translation endpoint that could swap between Llms and ML or something.

Actually I do agree with this change because I don't like it as I'd prefer if we thought this through and had some plan for operation types but it's sometimes good to do things organically and see what happens. I do wonder if operation types are going to morph into something where there are tons of different specific common functions that you wsnt to swap out (an alt text generation endpoint).

🇬🇧United Kingdom yautja_cetanu

Could you provide any insights into why you wanted a new operation type?

Its possible that the hook alter is good but the abstraction layer stops being one if every provider just has it's own operation type

🇬🇧United Kingdom yautja_cetanu

Hi Poker10, thanks for all your issues! They are very helpful. There are a number of things you've raised here. I'd seperate them out into three issues:

  • How do we let the end-user know the scope of the AI Assistant in Drupal CMS?
  • How do we make sure that the AI Assistant always uses the correct links.?
  • How can we use the AI Assistant to help you install, configure and make use of new models?

Regarding the first - Scope
The purpose of the AI Assistant in Drupal CMS is razor focused on helping use use the AI Agents for Content Types, Fields on Content Types and Taxonomy. It's also aimed at marketeers and sitebuilders with little Drupal experience, not developers at all. (We have another sub-module in the AI_Agents module for creating a form that allows you to mass create fields that is aimed at helping developers, but we think the chatbot is unlikely to be a good UI for developers interacting with AI compared to drush or specific forms).

I was thinking of specifically saying this in the opening message of the Chatbot but the problem with that is that it kind of scares the marketteer is we tell them the limitations as they won't know what "Content Types" are. So we have some ideas of how to handle this for 1.1 (For example having it so that the chatbot can only change fields for the entities of the page you're currently on so its more contextually aware and more focused).

Therefore it is not build to try and resolve your much more advanced issue of "how to use advanced aggregation?". So fixing specifically that issue isn't our priority right now. We do have plans for it (Creating a Project Browser AI Assistant). And are exploring them internally to decide if that will be a focus as it relies on a lot of moving parts and people. (More info below)

Re: Links

We noticed a lot of the hallucinations were due to the links, especially the assistant when giving relative URLs would regularly miss out the / first which caused issues and so we've tried to solve it. I think links are so important for 1.1 we need to build an AI Agent or some deteterministic validation to check every link and validate it.

We tried to solve it with prompt engineering but I think we need to do more.

Re: Advanced Aggregation and AI helping with modules.

Paul is correct that the specific thing you are asking it, isn't what we currently support and can solve. He is right about how those links are generated. As a result to get correct links there are three ways we can resolve it:

Train the model based on the new accurate links (For OpenAI and the bigger models this is too expensive)
Train a small opensource model on the specific links (This is more likely to work but for constantly changing links on a live site also not going to solve it)
Provide the information about the links directly into the prompt we give the AI which we call "In-context Learning" - This is the real solution to the problem.

As a result what we would do is use the AI Search module, or bake it into the drupal.org project browser search itself (or in the short term provide our own new source plugin for project browser). To take your query, go and search for "advanced aggregation" and then get some results that we provide an AI LLM, that will either find the module in question, present options to the end-user or find similar modules and present those options. Then we need to also have another database of drupal documentation that the AI Agent can search to find info about "How to config the "advanced aggregation" module. This would be another thing maybe Drupal.org would provide or maybe something else.

We would then tell the AI to ONLY use information from the context we've given it and never your own learning. Hallucination with this method is still possible but greatly reduced. We have a demo of this working, but quite a bit more work needs to be done.

Summary

Your proposed resolution is a good one: Verify links posted in the chat and do not provide links to D7 documentation.
Also we need the AI Agent to know the Drupal Version you're on (at least the top level if not the specific version from a security point of view).

🇬🇧United Kingdom yautja_cetanu

📌 Change instructions to add a review stage to the assistant Active Issue to implement the changes to the instructions.

Note: Catch, the Details that you can see in the issue above which are "Generating a list of 'created x type with y label' type messages might not be as fluid but it won't misrepresent what happens."

Still exist.

  • The list is generated and stored in a log so devs can use it for debugging.
  • The AI is writing its recap summarising the details.
  • The details is a setting in the module. So we can turn it on if we want to very easily. A siteadmin can turn it on if they want it. We're just turning it off by default for Drupal CMS 1.0.

For Drupal CMS 1.1 we will be meeting weekly with a UX team to figure out how to improve this further which can include:

  • The generated list of details being written more effectively.
  • Expanding "Blueprints" so that preview step generates a list of deterministic commands that are implemented with an explicit button press (instead of the AI agents implementing it).
  • Using more Javascript + Streaming so that the AI manually implements everything before your eyes. (It works well in XB but a bit harder with fields).
  • Workspaces use fields.
🇬🇧United Kingdom yautja_cetanu

Conditional will be 1.1

Production build 0.71.5 2024