Contrib.social

Iteration agent on created components

Open on Drupal.org →
Created on 25 June 2025, about 2 months ago

Background

We have been playing around manually with the Agent Explorer and the component creator agent to see if a subagent that looks at an image of an initial component and look at the state of the component will generate better results, and with three tests its obvious that it does.

One example:

We start with this image and ask the agent to generate it:

The first iteration as the agent works at the moment generates this:

We then loop this screenshot and the original image and ask a sub agent to look and describe what is needed to change. It comes up with:

### Key Changes Needed for testcomponent

1. **Background**
- Add a blue abstract gradient with glowing/light overlay effects using custom CSS. Keep Tailwind's bg-black as a fallback.

2. **Typography & Layout**
- Split the large heading into two lines with stronger sizing and spacing.
- Add a glowing or outer shadow/text-shadow effect to the hero text for emphasis (custom CSS).
- Increase spacing between all major elements.

3. **Top Badge**
- Update the "NEW" badge with a more vivid blue and subtle shadow/glow (custom CSS).
- Increase badge and padding roundedness to look more pill-like.

4. **Button**
- Make the button more pill-shaped, larger, with a blue glow/inner shadow (custom CSS).
- Lighter button font color.

5. **Supporting Text**
- Subheadline should be slightly lighter, add subtle blue tint if possible via CSS.

We then feed this into the agent again and tell it to change this, and it also give the two images and it comes up with:

Add another step and its at

So its clear that this improves the results.

Overview

We should build this in so the agent automatically can loop over its results until its happy. There are some questions that needs to be answered though:

We can do this via the frontend, since the iframe that previews the component is on the same origin - a tool like html2canvas should work. This however means that the loop or instruction has to be started over or triggered via the frontend, instead of the normal agent loop.

One option is that the agent returns a value of how well the new component fits the screenshot and if its under a threshold and it didn't reach max loops, it just sends it back for another iteration.

Since it costs money another solution is that after you generate a component from a prompt or image, that there is a button in the chatbot or on the preview that says something like "Improve quality" that runs another loop.

We can also do this via backend, but then we need an endpoint where we can render the unsaved component. On top of that we need to have headless chrome or somehing similar on the backend that can take screenshots, alternate a service that can run a proxy so it can work against your local machine.

The advantage with backend generation is that you will be able to create components over MCP (and through it Claude Code, Roo Tools, Cursor etc.) and also create a starterkit modules, that can pre-generate 10-15 components for you when you start a new website.

Proposed resolution

  • Figure out which, if any solution is feasible to deploy. We will also have a way where agents can get supplemental features via recipes in AI 1.3.0, so it doesn't have to ship with XB AI core.
  • Experiment with the html2canvas and XB to verify that it works.

User interface changes

TBD

✨ Feature request
Status

Active

Version

0.0

Component

AI

Created by

🇩🇪Germany marcus_johansson

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @marcus_johansson
  • Comment about 2 months ago →
    🇩🇪Germany marcus_johansson

    html2canvas tested with a manual button against the Experience Builder iframe and the results are not 1-to-1, so its not a solution that is viable.

    vs

    Its also stated in the documentation

    Why doesn't CSS property X render correctly or only partially?
    As each CSS property needs to be manually coded to render correctly, html2canvas will never have full CSS support. The library tries to support the most commonly used CSS properties to the extent that it can. If some CSS property is missing or incomplete and you feel that it should be part of the library, create test cases for it and a new issue for it.

    I will do the headless chrome tool, because that is needed for other reasons anyway, the question then is how it can access the component via localhost. Will try to figure this out.

  • Comment about 2 months ago →
    🇩🇪Germany marcus_johansson
  • Comment about 2 months ago →
    🇩🇪Germany marcus_johansson
  • Merge request !1199Draft: Resolve #3532296 "Iteration agent on" → (Open) created by marcus_johansson
  • Pipeline finished with Failed
    about 2 months ago
    Total: 1089s
    #532126
  • Pipeline finished with Failed
    about 2 months ago
    Total: 1559s
    #532202
  • Pipeline finished with Failed
    about 2 months ago
    Total: 1410s
    #532264
  • Pipeline finished with Failed
    about 2 months ago
    Total: 1197s
    #532400
  • Comment about 1 month ago →
    🇬🇧United Kingdom yautja_cetanu
  • Comment about 1 month ago →
    🇬🇧United Kingdom yautja_cetanu

    Follow up

    • Need to turn it into a proper agent now file upload hasn't happened.
    • Make it so we can swap out the back-end service for looking at the images.
    • Make it so we can configure the number of loops
  • Comment about 1 month ago →
    🇬🇧United Kingdom yautja_cetanu
  • Comment 19 days ago →
    🇩🇪Germany marcus_johansson

    So, the current setup for anyone taking over:

    There are three components to this agent:

    1. The preview controller.
    This is a controller that can preview the latest version of a JS component rendered within the frontend theme. This takes parameters that can be set with width and height in percentage or pixels. The reasoning is so you can request the component rendered with the frontend theme with the exact same measurements, so it can compare.

    2. The Rate XB Components tool.
    Currently it takes the file id and the component id, and uses Chromium to visit the controller above and take a screenshot and then it asks inside the tool to rate this from 1 to 10 how similar they are and what improvements can be made and returns these.

    Pre-requirements:
    This solution needs chromium installed can chrome-php.

    What is missing:
    On the controller we should add a possibility to have one off permissions. This means that the agent should be able to request and get a authentication code that is one-use only. Let me see if I can fix this.

    The tool should be possible to move into a tool that a sub-agent is using instead now that we can use images.

    The last part is to reiterate on the agent over and over until you get a result that is over N within A amount of loops. Rating 6 is something that worked quite well with the prompt above. I had that working in long timeout format, but with the iteration issue this should be possible to see over and over.

  • Comment 19 days ago →
    🇺🇸United States Kristen Pol Santa Cruz, CA, USA

    Switching to the correct tag

  • Comment 17 days ago →
    🇮🇳India Akhil Babu Chengannur
  • Comment 13 days ago →
    🇮🇳India Akhil Babu Chengannur

    Currently, the code component creation agent returns the component data (id, js, css etc) in JSON format and based on that, the actual component is created from the UI.
    But to review the component on the fly before sending the final output, the component must exist. Means, we should create it directly from the backend.

  • Comment 6 days ago →
    🇮🇳India Akhil Babu Chengannur
  • Comment 6 days ago →
    🇮🇳India Akhil Babu Chengannur

    akhil babu → changed the visibility of the branch 1.x to hidden.

  • Pipeline finished with Failed
    6 days ago
    Total: 968s
    #569957
  • Comment 6 days ago →
    🇮🇳India Akhil Babu Chengannur

    As mentioned in #12, component creation is currently happening from the UI. For the review to work, the component must be created from the backend. However, in that case, the CSS and JS will not be compiled properly, since the compilation happens in the frontend and the compiled CSS and JS are then sent to the backend. Because of this, the component will not be rendered in the preview controller.

    Also, when Chromium is used, the component is rendered as seen by an anonymous user. XB does not apply styles for anonymous users. The Create/Edit XB page content permission is required for this. Even if the permission is granted, the issue described above still prevents the component from being previewed.

  • Comment 5 days ago →
    🇬🇧United Kingdom yautja_cetanu

    For people not into AI basically we need to find a way that we automatically generate an accurate screenshot of a componenet.

    - The componenet is built in react by typing javascript, css etc.
    - We need some way we can take the componenet we see rendered and automatically take a screenshot without the user having to press screenshot screens.
    - We need to send that screenshot to a server side API for processing (in this case an AI agent looking at it) whilst thinking carefully of any security implications.

contrib.social Blog FAQ Discussions
Production build 0.71.5 2024