Looks like hallucination. Is it happening consistently?
Here is how it is supposed to work
If the prompt is in the format "Create a template for the homepage of an ice cream shop" -> Create the page layout without header and footer. Components should be mainly placed in the 'Content' region. It can also place components in other regions like Sidebar if enabled.
If the prompt is in the format "Create a template for the homepage of an ice cream shop with a header and footer" -> Create the page layout with header and footer. Header and footer components will be placed in corresponding regions if available . Else all components will be placed in the content region
The schema related error mentioned in #13 have been resolved now.
Closing this issue as the schema changes have been pushed in https://git.drupalcode.org/project/canvas/-/merge_requests/331/diffs?com...
Updated the schema of theme region settings config and the test is passing now. Thanks
Tests are up now. This is ready for code review.
Please note that this issue still needs some UI fixes so that the chat history element would appear properly on the explorer (see #8). I am not sure which tag should be used for that.
This issue would need some css fixes as well as the chat history element is not appearing properly in the explorer. The borders of input fields and dropdowns are not visible
Added the chat history element to the Chat explorer.
Testing steps
Start with demo umami profile
Install Tools module (
https://www.drupal.org/project/tool →
)
Install ai_api_explorer, Tool - Content, Tool - AI Connector
Setup an AI provider
Go to /admin/config/ai/explorers/tools_explorer
Select List entities tool
Give 'node' as entity_type_id and 'article' as 'bundle'
Click Run function to see an output like this
Tool Plugin executed successfully: Returned 8 <em class="placeholder">Content</em>(<em class="placeholder">node</em>) entities out of a total 8
Output for results: [{"_metadata":{"id":"18","type":"node","bundle":"article","label":"Give your oatmeal the ultimate makeover","uuid":"29fe2f51-bbaf-4573-b04b-4c74ef32eaba"}},{"_metadata":{"id":"17","type":"node","bundle":"article","label":"Skip the spirits with delicious mocktails","uuid":"a5a13c1e-bd77-4757-bc2c-461d68b0b2d4"}},{"_metadata":{"id":"16","type":"node","bundle":"article","label":"Baking mishaps - our troubleshooting tips","uuid":"9911d8c4-d724-4ded-901d-8d3010152d56"}},{"_metadata":{"id":"15","type":"node","bundle":"article","label":"Let's hear it for carrots","uuid":"8576994b-c1f0-48b5-b39a-848acab1ee3a"}},{"_metadata":{"id":"14","type":"node","bundle":"article","label":"The Umami guide to our favorite mushrooms","uuid":"020ae3dc-7b1d-4e9b-aa88-7124422843b6"}},{"_metadata":{"id":"13","type":"node","bundle":"article","label":"The real deal for supermarket savvy shopping","uuid":"3e4811d2-e01e-40b5-9d32-f846aa2ea7a4"}},{"_metadata":{"id":"12","type":"node","bundle":"article","label":"Dairy-free and delicious milk chocolate","uuid":"1d5acfe0-f530-4c9c-84ec-03158dee5f8f"}},{"_metadata":{"id":"11","type":"node","bundle":"article","label":"Give it a go and grow your own herbs","uuid":"f3f18a48-e932-4c23-8a47-c6f4644c4683"}}]In word, Google docs or any other editor add the text 'The real deal for supermarket savvy shopping', Take a screenshot and download it
Now open the chat explorer (admin/config/ai/explorers/chat_generator)
In the chat history element, add messages in the following order
Message - 1
Role: User
Content: How many article nodes are there?
Message -2
Role: Assistant
Content: Let me use the correct tool
Add tool call with Tool call ID: tool__entity_list, Function input: {"entity_type_id": "node", "bundle":"article"}, Tool call ID: 1234
Message-3
Role: Tool
Content: [{"_metadata":{"id":"18","type":"node","bundle":"article","label":"Give your oatmeal the ultimate makeover","uuid":"29fe2f51-bbaf-4573-b04b-4c74ef32eaba"}},{"_metadata":{"id":"17","type":"node","bundle":"article","label":"Skip the spirits with delicious mocktails","uuid":"a5a13c1e-bd77-4757-bc2c-461d68b0b2d4"}},{"_metadata":{"id":"16","type":"node","bundle":"article","label":"Baking mishaps - our troubleshooting tips","uuid":"9911d8c4-d724-4ded-901d-8d3010152d56"}},{"_metadata":{"id":"15","type":"node","bundle":"article","label":"Let's hear it for carrots","uuid":"8576994b-c1f0-48b5-b39a-848acab1ee3a"}},{"_metadata":{"id":"14","type":"node","bundle":"article","label":"The Umami guide to our favorite mushrooms","uuid":"020ae3dc-7b1d-4e9b-aa88-7124422843b6"}},{"_metadata":{"id":"13","type":"node","bundle":"article","label":"The real deal for supermarket savvy shopping","uuid":"3e4811d2-e01e-40b5-9d32-f846aa2ea7a4"}},{"_metadata":{"id":"12","type":"node","bundle":"article","label":"Dairy-free and delicious milk chocolate","uuid":"1d5acfe0-f530-4c9c-84ec-03158dee5f8f"}},{"_metadata":{"id":"11","type":"node","bundle":"article","label":"Give it a go and grow your own herbs","uuid":"f3f18a48-e932-4c23-8a47-c6f4644c4683"}}]
Tool call ID reference: 1234
Click Ask the AI button, the model should return an output like "There are 8 article nodes in total."
Now add two more message to the history
Message-4:
Role: Assistant
Content: There are 8 article nodes in total.
Message-5:
Role: User
Content: What is the id of the article with this title?
Upload the image with the article title created earlier, in the file field
Click Ask the AI button again. The model should output "The article with the title "The real deal for supermarket savvy shopping" has the ID: **13**."
As a developer, having such granular control is always cool.
Suppose I need to change the description of a tool at the third loop. So I might create an 'AiAgentProcessor' like this:
/**
* @AiAgentProcessor(
* id = "modify_tool_description",
* label = @Translation("Modify tool description"),
* description = @Translation("Change the tool description on a specific agent loop."),
* stages = {
* "pre_request" = 0,
* },
* )
*/
The processor could then be enabled for any agent from the UI, and it can provide a form to select the tool, loop count, and new description.
Is this roughly how you're imagining it, or do you see this working differently?
This issue builds on the work done in 🐛 Canvas AI: CanvasAiComponentDescriptionSettingsForm redirects to 'canvas.api.config.list' after submission Needs work and depends on it being merged first.
Here are the changes pushed in the last commit.
- Added a Detailed description field to all components in the form.
- The get_component_context tool returns the ID, name, and description of all components.
- The get_metadata_of_components tool returns the ID, name, detailed description, props, and slots of the components.
in https://git.drupalcode.org/project/canvas/-/merge_requests/18/diffs?comm..., I have added the following
- A test for the theme settings form
- Updated the existing test of set template data tool to cover complex scenarios
- Updated the schema of canvas_ai.theme_region.settings config as it was incorrect
The functional test to validate the form was throwing config validation errors like 'Schema errors for canvas_ai.theme_region.settings with the following errors: canvas_ai.theme_region.settings:region_descriptions.stark missing schema ' upon saving the form. I had to add protected $strictConfigSchema = FALSE; to disable strict config schema validation. This error does not happen on manually saving the form from. So not sure why the test was throwing the error
from 🐛 Canvas AI: CanvasAiComponentDescriptionSettingsForm redirects to 'canvas.api.config.list' after submission Needs work
We also need additional tests to verify:
- When a component/prop/slot description is overridden using the form, the tool should return the overridden description.
- When components from a particular source (e.g. Blocks) are disabled using the form, the get_component_context tool should return only the components from the enabled sources.
I have added the following tests
1. CanvasAiComponentDescriptionSettingsFormTest.php – verifies that the form title is rendered correctly and that form validation and submission behave as expected.
2. CanvasAiPageBuilderHelperTest.php – checks whether the component list is properly cached and ensures that the cache is invalidated when components are updated.
While creating the CanvasAiComponentDescriptionSettingsFormTest, I encountered the following error when submitting the form in the test:
Schema errors for canvas_ai.component_description.settings with the following errors: canvas_ai.component_description.settings:component_context.block missing schema in Drupal\Core\Config\Development\ConfigSchemaChecker->onConfigSave() (line 98 of core/lib/Drupal/Core/Config/Development/ConfigSchemaChecker.php).
This error does not occur when manually saving the form from the UI.
To work around this in the test, I had to add: protected $strictConfigSchema = FALSE; in the test. Will create a followup issue to check this further.
We also need additional tests to verify:
- When a component/prop/slot description is overridden using the form, the tool should return the overridden description.
- When components from a particular source (e.g. Blocks) are disabled using the form, the get_component_context tool should return only the components from the enabled sources.
These tests can be incorporated along with 📌 Simple approach to bringing advanced metadata into Canvas AI Active , as that issue modifies the output of the get_component_context tool.
I have updated the existing get_component_context tool which is given as a default information tool to the Page builder agent, to return only the id, name and description of all components, in the following yml format
Blocks:
block.system_menu_block.footer:
name: Footer
description: Footer
block.system_menu_block.main:
name: 'Main navigation'
description: 'Main navigation'
block.system_messages_block:
name: Messages
description: Messages
'Single-Directory Components':
sdc.space_ds.space-accordion:
name: 'Space Accordion'
description: 'Space Accordion'
sdc.space_ds.space-alert:
name: 'Space Alert'
description: 'A customizable alert component for displaying different types of status messages.'
sdc.space_ds.space-button:
name: 'Space Button'
description: 'A customizable button with options for icon, text, size, style, and state.'
'Code Components':
js.hero_banner:
name: 'Hero Banner'
description: 'Hero Banner'If the user has not used the CanvasAiComponentDescriptionSettingsForm to override the component, prop, or slot descriptions, then:
- For SDCs, the description defined in each component’s YAML file will be used. If no description is provided, the description will default to the component name.
- For Code Components and Blocks, the description will default to the component name.
If the CanvasAiComponentDescriptionSettingsForm has been used to override the descriptions, then those overridden descriptions will be loaded for all components.
A new tool, get_metadata_of_components, has been created. It accepts an array of component IDs and returns a single YAML response containing detailed metadata for each component.
- For SDCs and Code Components, this detailed metadata includes props and slots.
- For Blocks, the output remains the same as the output of get_component_context.
For example, if the input is
[block.system_menu_block.footer, sdc.space_ds.space-button, js.hero_banner], then the output of get_metadata_of_components tool would be
block.system_menu_block.footer:
id: block.system_menu_block.footer
name: Footer
description: Footer
sdc.space_ds.space-button:
id: sdc.space_ds.space-button
name: 'Space Button'
description: 'A customizable button with options for icon, text, size, style, and state to fit different interactions.'
group: Atoms
props:
icon:
name: Icon
description: 'The image used as an icon within the button.'
type: object
default: { src: /themes/contrib/space_ds/components/01-atoms/space-button/images/time.png, alt: 'Boring placeholder', width: 50, height: 50 }
text:
name: 'Button text'
description: 'The text displayed on the button.'
type: string
default: 'Contact Us'
required: true
url:
name: URL
description: 'The action URL that the button links to when clicked.'
type: string
default: 'https://www.qed42.com/contact'
required: true
variant:
name: Variants
description: 'The visual style of the button (e.g., primary, secondary).'
type: string
default: primary
enum: [primary, secondary, tertiary, ghost, link]
size:
name: Size
description: 'The size of the button (e.g., small, medium, large).'
type: string
default: small
required: true
enum: [small, medium, large, xlarge, xxlarge, xxxlarge]
button_corners:
name: Corners
description: 'The style of the button corners (e.g., rounded or square).'
type: string
default: rounded_half_circle
enum: [rectangle_corner, rounded_half_circle]
button_type:
name: Type
description: 'The type of button, determining whether it shows only an icon, only text, or both.'
type: string
default: only_text
required: true
enum: [only_icon, text_with_icon, only_text]
icon_alignment:
name: 'Icon alignment'
description: 'The alignment of the icon relative to the button text (left or right).'
type: string
default: right
enum: [only-icon, left, right]
expanded:
name: Expanded
description: 'Whether the button should be full width or auto width.'
type: string
default: 'no'
enum: ['yes', 'no']
button_state:
name: State
description: 'The state of the button (e.g., focused or disabled).'
type: string
default: focused
enum: [focused, disabled]
classes:
name: 'Class names'
description: 'Custom CSS class names (separate by spaces), applied when custom style is enabled.'
type: string
default: null
slots: 'No slots'
js.hero_banner:
id: js.hero_banner
name: 'Hero Banner'
description: 'Hero Banner'
props:
headline:
name: headline
description: headline
type: string
default: 'Welcome to Our Site'
format: ''
enum: ''
subtext:
name: subtext
description: subtext
type: string
default: 'Discover amazing content and connect with our community.'
format: ''
enum: ''
ctaText:
name: ctaText
description: ctaText
type: string
default: 'Get Started'
format: ''
enum: ''
ctaLink:
name: ctaLink
description: ctaLink
type: string
default: /
format: uri-reference
enum: ''
intent:
name: intent
description: intent
type: string
default: primary
format: ''
enum: [primary, secondary]
slots: { }
The page builder (and template builder, once
✨
Introduce AI Agents and tools to create entire page templates using available component entities
Active
gets merged) would use this tool to get detaield info about any components.
If the current approach looks fine, I could start incorporating @jamie's suggestions in #10
Started workin on this along with 📌 Simple approach to bringing advanced metadata into Canvas AI Active
Thanks for reviewing this @narendrar. While reworking this, I feel that the sub-requests added in ✨ Use subrequests when getting component list in getAllComponentsKeyedBySource Active might not be needed. This sub-request gets fired every time the form is loaded and submitted. Even if we cache the results, the sub-request will get fired at least once when the user loads the form for the first time. Due to this, the form title won't get displayed as it interferes with the form request.
I feel the better approach is to bring back the original logic of invoking the controller method here, do the necessary calculations, and then cache the result. This eliminates the sub-request and thus fixes the issue with the form.
Regarding the tests, I think it makes more sense to add a test for the get_component_context tool to ensure that
1. If user doesnt use the form, this tool returns the components with their default descriptions
2. If user uses the form to override component descriptions, the tool returns those.
That can be it's own separate issue. What do you think?
I have pushed some updates:
1. Previously, the template builder was responsible for generating headers and footers. Now:
- If a user asks to create a page with a header and/or footer, the template builder agent will generate them.
- But when a user creates a page first and then asks to add a header or footer, it will be handled by the page builder agent.
Because, the page builder agent can now add components to multiple regions or positions at the same time, a prompt like “Add a header and footer for a page about comic books” will work. The components will be placed in the appropriate regions.
2. With the above change, the `SetTemplateData` tool used by the template builder agent no longer requires the reference_nodepath argument.This argument was originally needed to place footer components in the correct position. Since the page builder now handles this, the argument can be removed.As a result, the canvasAiPagebuilderHelper::processComponentsBelow() method can also be removed.
3. Updated the orchestrator prompt. Even though it may look like many changes, most of them are spacing and structural improvements, along with new template-builder–specific instructions.
4. Updated the polling messages as follows, since these are more accurate functionally:
'canvas_page_builder_agent' => $this->t('Finding components to place'),
'canvas_template_builder_agent' => $this->t('Designing the page'),
This response is triggered from the following section of the component agent's prompt.
CRITICAL FLOW GATEKEEPER:
- **TERMINAL EDIT CHECK:** If the user's request is an instruction to **modify, edit, change, or update** a component and the context variable `[canvas_ai:selected_component]` is null, this is a **TERMINAL FAILURE**.
- You **MUST STOP** all further processing immediately.
- Your **ONLY** response MUST be the exact string: **"The 'edit component' functionality is only available on component edit route."**
- Do NOT add any extra text, explanation, apology, or tool output.
- Do NOT proceed to any other flow or use any tool.Within the prompt we are asking the agent to check if the variable `[canvas_ai:selected_component]` is null. But as [canvas_ai:selected_component] is a token, its value gets replaced before the prompt is sent to the agent. If the component name is 'hero_banner', then the instruction becomes
- **TERMINAL EDIT CHECK:** If the user's request is an instruction to **modify, edit, change, or update** a component and the context variable <em>hero_banner</em> is null, this is a **TERMINAL FAILURE**.
Which does't make sense.
We should leverage the [canvas_ai:get_verbose_context_for_orchestrator] token here. If user is editing a componnet, then the tokens value becomes
User is now in the code component editor, viewing a code component with id hero_banner
So within the instructions of we can add an instruction like
'Validate the component data before attempting an edit. Only proceed if the user is currently in the code component editor, and ensure you use the exact id from the component data.'
then at the end of the prompt we can add
##Component data
[canvas_ai:get_verbose_context_for_orchestrator]
Verified this with @narendrar. For OpenAI and Anthropic models (except Claude 3.7 Sonnet), tool calls are executed in parallel. For example, when both the title and description are empty, the orchestrator invokes the page_builder/template_builder, title_agent, and `metadata_agent` simultaneously.
However, with Claude 3.7 Sonnet, the tool calls happen sequentially. We confirmed this by creating a simple taxonomy agent that adds taxonomy terms to a vocabulary using the `modify_taxonomy_term` tool. When asked to add more than one term to a vocabulary, it consistently invokes the tool multiple times sequentially, even when explicitly instructed in the prompt to use the tools parallelly.
Also, When the orchestrator is invoked for the first time, the model correctly indicates that it will generate both a title and description, but once it invokes page_builder or template_builter, the execution stops
Rgarding #65, this might be specific to anthropic claude models. I am using openai and title and descriptions are getting generated correctly.
Placed component are not displayed if text color matches background.
The component description settings form can be used to add instructiins specific to each components (Eg: If there is a theme prop with light and dark options, when to use the 'light' prop and when to use 'dark'. The agent will consider these instructions when generating the template
The final message from the orchestrator is currently displayed at the top. The message from the last sub-agent that was invoked is being added to the chat history, but ideally, the orchestrator’s response should appear at the bottom and be added to the history instead.
This would require refactoring once 📌 CanvasBuilder::render() logic around tool specific tools response is unclear Active lands
YAML decoding used to fail in some random scenarios when there were special characters in the component structure generated by the agent. That's why JSON encoding/encoding was added for the setAIgeneratedComponentStructure tool. The YAML decoding of the tool result was a bug, and I had fixed it in this MR. But these changes will no longer be needed once 📌 CanvasBuilder::render() logic around tool specific tools response is unclear Active lands.
Verified page builder functionality manually with these changes and did not find any issues. +1 RTBC
Thanks for the detailed review. I have pushed the required changes. The template builder prompt is also updated as 📌 Canvas AI: Refine the orchesrtrator prompt Active is now merged. It now focuses only on creating the page. Page tite/metadata generation and Entity type validation is now handled by the orchestrator. So, the frequency of occurrence of the issue mentioned in #58 should also decrease.
Thanks for the patch @aalin. I have included some visibility settings as well so that taxonomy term related form elements wont be there when user select's 'Language' option
akhil babu → made their first commit to this issue’s fork.
Currently, only the last two messages are sent as history to the orchestrator, which affects the overall user experience.
See the attached videos for reference. When asked to create a CTA for a tech journal website, the orchestrator asks whether a code component should be created or an existing component should be added (meaning the page builder should be used). When asked to create a code component, the component agent fails. So I asked the agent to "add an existing component then." The agent then responds with:
"Please let me know what type of existing component or section you would like to add to your page, and any specific details or placements you have in mind."
This happens because it doesn’t have access to the complete chat history.
Another issue is that intermediate status messages from sub-agents are also included in the history. For example:
When you ask the agent to "Create a button code component with a red background," the component agent is invoked. Once the component is created, it outputs:
"I have created a new 'Red Button' component with customizable button text and a red background. Let me know if you need any further adjustments!"
When the orchestrator receives this message, it also outputs something like:
"A new 'Red Button' component has been created with customizable button text and a red background. You can now use and adjust the button text as needed."
Now, when the user asks to change the color to blue, the message history sent to the orchestrator looks like this:
This includes the outputs of both the component agent and the orchestrator, but not the initial user message.
[
{
"role": "ai",
"html": "<div style=\"margin-top: 10px;\"><div style=\"display: flex; align-items: center; padding: 8px; background-color: white;\">\n <span style=\"margin-right: 8px;\"><span class=\"aiCompletedIcon\"></span></span>\n <span style=\"font-weight: 400;\">Thinking</span>\n </div><div style=\"padding: 8px; background-color: white; font-size: 14px; line-height: 1.26;\">\n I have created a new \"Red Button\" component with a customizable button text and a red background. Let me know if you need any further adjustments!\n </div><div style=\"display: flex; align-items: center; padding: 8px; background-color: white;\">\n <span style=\"margin-right: 8px;\"><span class=\"aiCompletedIcon\"></span></span>\n <span style=\"font-weight: 400;\">Generate a component</span>\n </div><div style=\"padding: 8px; background-color: white; font-size: 14px; line-height: 1.26;\">\n A new \"Red Button\" component has been created with a customizable button text and a red background. You can now use and adjust the button text as needed.\n </div></div>"
},
{
"role": "ai",
"text": "A new \"Red Button\" component has been created with a customizable button text and a red background. You can now use and adjust the button text as needed."
},
{
"role": "user",
"text": "change its color to blue"
}
]I think the sub-agent output can be ignored in the history since the orchestrator already returns a similar message. Instead, the history should look like this:
[
{
"role": "user",
"text": "Add a button with red color"
},
{
"role": "ai",
"text": "A new 'Red Button' component has been created with a customizable button text and a red background. You can now use and adjust the button text as needed."
},
{
"role": "user",
"text": "change its color to blue"
}
]HI All, We are planning a webinar on 29 October 2025, to introduce Drupal AI to a wider audience and encourage contributions. This issue already has an existing patch, but its been in active state for 3 months. I am assigning it to myself so I can test the patch, make any necessary corrections, and raise a Merge Request during the session.
Updated the orchestrator prompt to fix #57. While testing it, I observed another issue
I am using the OpenAI provider to test the prompt:
“Create a header for a restaurant site with a logo, search bar, and main navigation menu.”
The GPT-4.1 model often doesn’t delegate the task to the corresponding sub-agent. It simply says “Your request will be processed.”
I tried the same prompt with GPT-4.1-mini, but this time it refused to create the header, saying that the entity type is not a Canvas page, even though it actually was.
I created 📌 Canvas AI: Refine the orchesrtrator prompt Active to move the entity_type check and title/metadata generation logic to the orchestrator so that the Template Builder and Page Builder agents can focus only on their core tasks. This should at least help fix the latter issue.
The issue with 4.1 is a bit more complicated . I've often observed that it claims it’s going to use a tool, but never actually does.One option is to use the check added in ✨ Add possibility to check if a tool was run Active . However, that may not be the optimal solution, because we can't always guarantee that the orchestrator or any sub-agent should use a specific tool.
For example:
- The user might just ask a question like “What can you do?” - no tool is required.
- Or the user might say “Add an accordion component to the page”, but if there is no accordion component available, the agent shouldn’t invoke a tool in that case either.
Created https://www.drupal.org/project/canvas/issues/3554026 📌 Canvas AI: Refine the orchesrtrator prompt Active as per #11
Created https://www.drupal.org/project/canvas/issues/3553998 📌 Canvas AI: Template builder agent should be aware of existing components in regions Active as a follow-up as per #52.
Verified using card components with required and optional image props, and it works as expected.
Thanks for the feedbacks narendrar,
Should create title and metadata if not exist for page as done for page builder earlier
- Done
Create a template for the homepage of an ice cream shop website: It is observed that it did not place button in Empty button slot for hero component. Also content slot is empty in spacer and desired content is placed below spacer.
This can be enforced from the component description settings form.
Create a template for the homepage of an ice cream shop website, with proper header and footer: If content in header/footer region exists, it places duplicate components also.
If user explicitly asks to generte headera and footer, agent will generate contents even if those regions already have components. I think we could open a followup to optimize this behaviour
Now that 🐛 XB AI: User selection should be taken into account while building page Active is merged, this needs some updates
So, if a user selects a component that has slots and asks the agent to “add a card to the slots of this component,” the agent will go ahead and add the card, even if the card isn’t a suitable child component for that slot.
The suitable child components for each slot can be specified in the slot description from Canvas AI Component Description Settings. But that’s only taken into account when the agent designs the component structure.
For example, in the Starshot demo theme, if we ask the agent to add a hero banner suitable for the landing page of a streaming platform, it will add a hero banner with Paragraph and Button components in the slots. But if a user manually adds a hero banner, clicks on it, and then asks the agent to add another hero banner to the slot, the agent will still go ahead and add it. It depends on how the user phrases the request.
That said, if such validation is really required, it should be possible with some prompt adjustments. But I think that can be handled as a separate issue.
Also, We need to pass proper keywords as instructions. Eg instead of Can you add some cards to show case the product highlights?, if we ask Using page builder can you add some cards to show case the product highlights? or Can you add some cards to show case the product highlights on this page?, orchestrator will pass the request to correct agent for selected component and it will work as expected for scenario 1.
The orchestrator agent invokes the page builder agent correctly only when user inputs are in the format “Add something to the page” or when inputs are explicitly prepended with “Using page builder, do something.”
A direct prompt like “Can you add some cards to showcase the product highlights?” often confuses the orchestrator. In such cases, it may invoke the component agent instead and create a code component.
Resloved the merge conflicts and verified in local. Moving back to RTBC
Thanks for the detailed analysis @tedbow . I have tested the latest changes in AiResponseValidator and confirming that its working when image data is passed in the below format
reference_nodepath: [0, 0]
placement: above
components:
- sdc.canvas_test_sdc.card:
props:
heading: "Delicious Cookie Selection"
content: "Indulge your sweet tooth with our mouthwatering variety of freshly baked cookies! Choose from chocolate chip, oatmeal raisin, snickerdoodle, and more—all made in-house every morning."
footer: "Order today for same-day pickup or delivery!"
loading: eager
sizes: "auto 50vw"
image:
src: /modules/contrib/experience_builder/tests/modules/canvas_test_sdc/components/card/balloons.png
alt: 'Hot air balloons'
width: 640
height: 427
I have updated the MR with following changes
1. Removed the previously added get_media_id tool and related changes as its no longer required
2. Updated the page builder agent's prompt so that it returns the image data properly
3. Updated Drupal\canvas_ai\CanvasAiPageBuilderHelper::processSdc to propely get the detault values for SDCs as described in #11
Out of all 4 sub agents of orchestrator the following only work with Canvas page
Drupal Canvas Metadata Generation Agent
Drupal Canvas Title Generation Agent
Drupal Canvas Page Builder Agent
The remaining agent is the Drupal Canvas Component Agent, which doesn’t care about the entity type. So I don't understand why we shouldn’t use the get_entity_info tool in the orchestrator and ask it to decline the request if the entity type is not canvas_page.
It has 2 Advantages
1. One API request can be saved, which also reduces cost. (For example, if the request is for a page building task, when the page builder agent gets invoked, the entire component context would get passed along with the system prompt, which is very costly.)
2. The user would get a faster response.
The only issue with this approach is when the user is exactly on the /canvas page, i.e., there is no entity_type value. In that case, the orchestrator could ask the user to create a page first before attempting to generate metadata, titles, or add components. Then later we can add new tools to agents to create 'Canvas page' entities programatically when entity_type is empty.
Significant changes have been made to the page builder agent in 🐛 XB AI: User selection should be taken into account while building page Active . Can we hold this issue until that gets merged?