Introduce AI Agents and tools to Build Pages from Single Directory Components

Currently there are 2 issues

1. The messages from the page_builder_agent are not rendering in the chatbot window.
2. Once a component is selected from the canvas, the screen automatically refreshes while typing the next message. It seems that a request is somehow being triggered to the endpoint /xb/api/v0/form/content-entity/{entity_type}/{entity}/{entity_form_mode}

I think we can limit scope of this issue to only create page using existing components.
Eg. "Create a page for our university website featuring a heading, a hero section, featured content, and news sections." should create the page with existing components.

Creating/editing of components using AI already exists and we should use those tools wherever required.

I tried testing the current MR, but the changes are not reflected on my local. Could I be missing something? Please see below.

The changes aren’t about creating or editing components. They’re focused on adding already existing components (Only SDC as of now) to the canvas with dynamic content to help build pages. For example: “Create a section showing three key features of Drupal” or “Add a button with the text ‘Read more.’”
In my opinion, creating entire page templates (with header, body, footer, etc.) can be handled in a separate issue after the current approach is reviewed and accepted, since we’ll need to reuse a lot of the existing logic for that.

Also, have you run npm run build after pulling the latest changes? If yes, are you seeing any errors in the console?

Forgot to run npm commands , now it is working as shown in Demo, Thanks.
We need to decide whether considering only the SDC component for now is acceptable.
Also, I have a question: What is the purpose of using 'default_information_tools' here instead of using the tool directly?

The page builder AI agent needs both the curent layout structure (Provided by the get_current_layout tool) of the page and the details of avaialble SDCs (Provided by get_sdc_info tool). Default information tools option helps to send these direct;y along with the Agent instructions as context. So it saves two extra AI calls.

Moving this back to Active for the following updates

1. Create a form that allows site builders to customize the description of SDC components and their props/slots. Upon submission, this description will be sent to the agent as context.

2. Extend the same form with a checkbox list that lets site builders choose which enabled component entities (SDCs, Blocks, Code components) should be included as context for the LLM. Currently only SDCs are allowed

I have tested this and it works great! Very impressed. I'm guessing there is another issue for editing components and their values?

I think in general, it would be more stable to input a structured function call with multiple levels of function calls, but that can be quite complex to setup and slower. If you feel that this works most of the time, its a faster way to do AI inference and to change to changing structures.

Some improvements (that might be out of scope for this issue):

The YAML structure should be as minimal as possible (maybe already is) - the wait time is 99% invested on generating tokens, and the less tokens it needs to generate the better.

Since we added the possibility to step through the loops it would be nice if the Orchestration Agent splits up the requests to the Page Builder Agent into loops even if this takes 3-5% extra time to produce, because then for a request like "Create me three components on the page for my pizza shop Marios Pizza", would create the first component and have a loader for that and you would see it, then it would start on the next etc. - so you see progress. 📌 Integrate incremental agent loop execution in XB AI Active will be needed for this.

Later adding so the system automatically creates a grid image of all components as example with very low resolution, and giving that to the agent in general, makes everything better. Then the agents will be able to see how the components look when its picking or if it might suggest you an component instead of createing a new one.

Obviously the descriptions of the components needs to be better, but that is out of scope for this issue.

Also as mentioned yesterday - if you need help with anything of this, let me know. Just write me on Slack.

Thanks for reviewing @marcus_johansson. I’ll go through your feedbacks and get back to you.

Thank you for the encouraging feedback @marcus_johansson.

I'm guessing there is another issue for editing components and their values?

Yes. Currenly only component addition is implemented. New issues will be created for updating/rearranging existing components placed in a page.
Eg: User clicks on a paragraph component and asks the agent to summarize/elaborate/replace it's content and props.

The YAML structure should be as minimal as possible (maybe already is) - the wait time is 99% invested on generating tokens, and the less tokens it needs to generate the better.

The front end only needs 2 options to add a component to the page
1. The component object
2. nodePath (An array represnting the position of the component: [0,0] -> First component, [0, 0, 0, 0] -> First component in the first slot of the first component 😬)

Initially, I tried asking the agent to generate the nodepath, but it started hallucinating when components had to be nested deep. So, in the current approach we ask the agent to provide a yml in the following format

    reference_nodepath: []  # Nodepath of any existing component placed in the page.(Agent gets it from the current layout json provided as context)
    placement: below                           # 'above' or 'below'
    components:
      - sdc.component.id:
          props:
            prop_name: "value"
          slots:
            slot_name:
              - sdc.nested.component:
                  props:
                    nested_prop: "value"
    message: "Concise 1-2 sentence summary of changes made"

and then generate the nodepath from this yml. The data for each component in the yml contains its id, prop names and values, slot names and child components.
The front end gets these details -> Loads the component objects based on ID -> Replaces the prop values and places each component at the specified nodepath.
So the yml contains only the required data.

Since we added the possibility to step through the loops it would be nice if the Orchestration Agent splits up the requests to the Page Builder Agent into loops even if this takes 3-5% extra time to produce, because then for a request like "Create me three components on the page for my pizza shop Marios Pizza", would create the first component and have a loader for that and you would see it, then it would start on the next etc. - so you see progress. #3531000: Integrate incremental agent loop execution in XB AI will be needed for this.

Something like this would definitely improve the user experience. I haven't checked 📌 Integrate incremental agent loop execution in XB AI Active yet, but I'm going to give it a try.

Most of the feedbacks have been addressed. Kindly review the changes again, Thanks

The components are not getting placed in the canvas when they are nested. Moving back to active to check this

Minor comment about permissions.

This looks good to me, just some nits/suggestions to address.

Also, there is one modules/xb_ai/src/test.yml file, which can be removed.

This feature has many functionalities, such as adding single/multiple components, order of placement, form to update description of components, etc., so I suggest that someone manually test and verify if all existing and this issue's functionality is working as expected or not.

Kernel tests can be written for the created components.

Also, when I tested this manually, I found the below issue:
First, I created a code component using the prompt "Could you create a hero banner component with some text, call-to-action button, and background colour red".
Then I added the following prompt to test this MR functionality: "Create a Hero banner showing three key features of Drupal and Create a section showing three key features of PHP"
It created a HeroBanner component and 2 sections - one for Drupal and one for PHP. This seems incorrect to me.

First, I created a code component using the prompt "Could you create a hero banner component with some text, call-to-action button, and background colour red".
Then I added the following prompt to test this MR functionality: "Create a Hero banner showing three key features of Drupal and Create a section showing three key features of PHP"
It created a HeroBanner component and 2 sections - one for Drupal and one for PHP. This seems incorrect to me.

Did it create the code component again? Also after generating the hero anner, was it added to the component library using the 'Add to components' option?
Code component generation and page building do not work together as of now. This means you cannot use a single prompt like: `Create a hero banner with text and a CTA, and place two banners on the page: one with content about Drupal and another with content about PHP.`

The orchestratior agent may sometimes delegate the task to the code component creation agent if the prompt contains words like 'Create'. So, to test page building, we should either start the prompt with 'Add/place a hero banner...' or at the end of the prompt add '..using the page builder agent'

Re #33, Yes, I first created the code component from the first prompt and then saved it to the component library. Then I used the second prompt and it did place the Hero Banner on the canvas, but with 2 sections. It should have placed one Hero Banner and one section as asked in the prompt.

Updated the code as per the PR feedbacks. Kindly review again after taking latest pull and importing configs, Thanks

Testing extensively AI capability, it's getting really impressive !

I found in the instructions, something incorrect regarding slots.

If a component has `slots`, you **MUST** fill those slots with valid, appropriate child components. **There can be no empty slots.** Decompose complex requests into a logical hierarchy of parent and child components.

LLM are not forced to fill slot, it's completely valid to have empty slot, I'm not even sure we can set slot to be required.

For example I have a heading component that has optional 'Badge' slot to add a badge at the end.

Getting this error sometimes

Error: Call to a member function getReadableOutput() on null in Drupal\xb_ai\Controller\XbBuilder->render() (line 219 of /var/www/html/web/modules/contrib/experience_builder/modules/xb_ai/src/Controller/XbBuilder.php).

Added Kernel tests to the MR.

There are some minor issues to address, but otherwise the code looks good to me. Manual testing of all relevant scenarios—both existing and new functionality—is still required.

Re #32, Two follow-up issues should be created and linked here.

Note: The PHPUnit tests are failing, but this does not seem related to the current issue.

Created ✨ XB AI: Centralized Management of Component Descriptions, Props, and Slots Active as per #32

Tested current MR with 🐛 AI-created new components cause loss of DeepChat message history Active . The final message from the page builder agent is not getting printed in the chatbot interface 😫 . We would have to create a followup issue for that as well

Re #41, This could be fixed here by moving operations logic outside of receiveMessage method.
Eg:.

      for (const handler of handlers) {
        await handler.handle(context);
      }
      
      // Process operations in background without blocking return
      if ('operations' in message && message.operations) {
        processOperations(message.operations);
      }
      
      return { text: message.message };

And then add logic in a separate function to handle operations. Eg:

const processOperations = async (operations: any[]) => {
  try {
    for (const op of operations) {
      if (
        op.operation === 'ADD' &&
        op.components &&
        Array.isArray(op.components) &&
        availableComponents
      ) {
        for (const component of op.components) {
          await delay(1000);
        }
      }
    }
  } catch (error) {
    console.error('Error processing operations:', error);
  }
};

Or, may be doing it in the way it is being done for other features. That is adding logic in getHandlersForMessage

        const handlers = getHandlersForMessage(message);
        for (const handler of handlers) {
          await handler.handle(context);
        }

I think second option is better to maintain consistancy.

I opened a follow-up for #37: 📌 Follow-up to #3530733: Allow for empty slots Active

I also opened a child issue regarding the impossibility to set link for a button component for example or src of an image.

🐛 Impossible to set a URL or SRC of an SDC with AI Active

Another issue I encountered was with SDC which has multiple slots.
If in the prompt, we specify to insert components inside a slot by providing it's name, the LLM will always insert the SDC into the first slot.
Following the advice of Akhil to better understand what's the going on, prompting something like insert a button in all slots of a specified component will do work.

A more problematic issue, sometimes LLM will set value of enums which doesn't exist, so it will trigger an error in XB to the impossibility to render the sdc

And not a bug, but more a feature request, today it's possible to insert SDC, but what about blocks and patterns (sections) ?
It will be a great addition if we were able to place a section, and let the LLM to fill existing props according prompt, should decrease unexpected results of llm

Thanks for the feedbacks @heyyo.
#45 was due to a bug in the code. I have fixed that in 37631920 . It should work now.

#44
Urls are correctly getting added if the prop type for the url field is string. Eg:

    url:
      type: string
      title: Url
      description: URL for the link button.
      examples:
        - 'https://drupal.org

It might not be working correctly for 'uri' type props. I think we could support that in 🐛 Impossible to set a URL or SRC of an SDC with AI Active

#46
Added enum validation in b17588fd . This increases the response time as it forces the AI to regenerate the entire response if it returns a non-existing enum value. But at least we shouldn't face a "component failed to render" error.

If enums prop is not required, maybe we can just removed non valid option, instead of regenerating the response again, also we can enter into infinite loop if LLM is stubborn 😄

Edit: Reverting the enum validation added in #48 as it breaks tests.

If enums prop is not required, maybe we can just removed non valid option, instead of regenerating the response again, also we can enter into infinite loop if LLM is stubborn

The max_loop setting in agents helps prevent them from going into an infinite loop. Simply removing the incorrect value might not be the best solution here, as I think we may still face the same "component failed to render" issue if the prop is required (though I haven’t tested this yet).

I think it’s better to handle this in a follow-up issue, where all of us can discuss our ideas for a proper fix.

My idea is to regenerate response only if prop is not required and invalid enum value

Created ✨ XB AI: Add enum validation Active

#41 is also fixed in the current MR. I think we could merge this now.

Manually tested; the happy path works as expected. Follow-up issues are created. This can be merged to unblock them and related work.
Moving it to RTBC.

Saving credit

Merged!

Automatically closed - issue fixed for 2 weeks with no activity.