--- AI TRACKER METADATA ---
Update Summary: Options all exist in different places, need to bring it together.
Check-in Date: MM/DD/YYYY (US format) [When we should see progress/get an update]
Due Date: MM/DD/YYYY (US format) [When the issue should be fully completed]
Blocked by: [#XXXXXX] (New issues on new lines)
Additional Collaborators: @username1, @username2
AI Tracker found here: https://www.drupalstarforge.ai/
--- END METADATA ---
Overview
- Create as a sandbox module now. Or can go into github is canvas needs to be hacked at all.
- This might be useful as a generic module for outside of canvas so that Drupal can take screenshots of the page.
- This is likely useful as an MCP endpoint for Drupal/ Canvas.
- It's possible this module needs to be in canvas itself as React may have inherently different ways of implementing the below.
- Create a module that provides the 3 different approaches as tools and any config/ setup required to get the more advanced tools (chromium php) working with the ability in the AI Agents to configure which one you use.
- Figure out any UI stuff in canvas we might need for firing off the tool (Maybe its automatic, maybe its a button?)
Proposed resolution
User interface changes
Background information:
came from this comment here:
✨
Iteration agent on created components
Active
I'll bring this up at the XB AI meetup but I think we should create another module for this to experiment on, could be a sandbox module, could be something else and maybe bring it back in. As I think we need to find out how to make something that physically works before figuring out the ideal UI.
There are three approaches:
- Use the html2canvas stuff in the example above - This is the simplest, and provides high resolution screenshots, but they are sometimes inaccurate (incorrect fonts, background colours, etc). - You can find a video and MR here:
https://www.drupal.org/project/canvas/issues/3532296#comment-16236352
✨
Iteration agent on created components
Active
- Use the browser screenshot tool - This requires the User to click a button giving permission for the screen to be shared with XB AI like when you share a screen in google meet. It provides accurate images but they are low resolution and grainy (which I think will be fine, AI doesn't seem to do better with higher res images as it takes up more tokens. TODO Andrew- Can you link to even your broken code somewhere?
- php chrome or playwrite, server side with a headless browser - This is the most likely to work method, and will be necessary if we use agents working in the background so we want to do this eventually. But its the hardest to host and make work. (Agents need to direct a browser). - TODO Marcus - Can you link to where your chromium php stuff exists?
I think we need to make a module that allows us to switch between the 3 above so we can try prompt engineering to get it to work well. With testing a review agent using chatgpt playground, we got the design to look pretty good after one iteration. As you can see above, in the canvas one, it didn't look good after one iteration (but that might just be the prompt). When we have a better idea of what works we can think about the different aspects of the UI we have to worry about:
- Accuracy of the Image - One reason why we went for making it so it uploaded the screenshot in the chat is that the screenshot isn't accurate sometimes. This gives the end-user the ability to see and understand why its not quite working. (But obviously if we can make the screenshots accurate then we don't need this.
- How long this takes - Review agents significantly increase the cost and time it takes to do anything. We may need some UI that allows people to choose (Like chatgpt has thinking mode vs not) or can we make it happen quickly?
- Why are these review agents happening and what prompts should fire them off or not? In the demo above we side-stepped it by giving the user a button. But the example of, using a figma screenshot and making an exact duplicate is unlikely to be a major way the chatbot is used, if that is our goal a dedicated screenshot to drupal tool is probably better. Its more likely they will upload images for inspirtional. So that would change things a little. (Maybe you don't want a review agent as you want user feedback quickly).
So my proposal is:
We create a module that allows you to pick between 3 different approaches and we can try the prompt engineering to figure out what is the best approach and then we can use this information to answer the UX questions above (eg break points etc).