AI -KR3 - Making experimenting feel safe.

Created on 12 November 2024, 13 days ago

Problem/Motivation

KR3 - Add an undo feature that lets marketers reverse agent actions -- evaluate design a UX for rollback functionality, and evaluate how much of Drupal's Workspace functionality we can reuse. Goal: end-users can feel safe experimenting with AI and we can take a bit more risk with AI agent development.

AI will probably not get every request 100% correct. We have a number of ways that we can try and mitigate this:

Proposed resolution

  • Workspaces - Make use of core Drupal workspaces so that AI can make changes to a specific workspace and the user can check everything and publish when they are happy or roll back to a previous version.
  • Have an "Undo" button next to something AI has done - (Complicated to show exactly what is being undone when there are a series of actions taken).
  • Provide logs and details of the specific actions the AI has taken in code. So that Drupal is report exactly what has happened (unlikely to make sense to the end-user).
  • Provide verbose descriptions of what AI has done generated by the AI including links so the end-user can check everything. (Possible chance to hallucinate)
  • Create a step where the AI describes what it will do in plain english and you have to say "Yes" or even click a button before it implements that step. (Again confusing when there are multiple steps required)
  • Blueprints - Provide a step where the AI tells you the exact JSON/ YAML it has generated to tell Drupal what to do that can be accepted before Drupal implements it. Much safer but unlikely to make sense to an end-user

One issue is Personas. Logs of what the code is doing is likely to appeal to developers but maybe it is too confusing for the ambitious sitebuilder or marketeer.

πŸ“Œ Task
Status

Active

Component

Track: AI

Created by

πŸ‡¬πŸ‡§United Kingdom yautja_cetanu

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @yautja_cetanu
  • πŸ‡¬πŸ‡§United Kingdom yautja_cetanu

    A couple of things from our initial investigations:

    1. Workspaces looks good but doesn't support field config yet which is essential so its unlikely to be ready for RC1

    Below is an AI Agent trying to add something to a vocab.

    It shows up as not yet pushed.

    2. We have verbose responses with links and details.

    Here it is medium levels of verbosity. It tells you the category but it doesn't tell you literally every term its going to make. It does give you links to places where you can check the work.

    Below is a details dropdown. It shows to a log of the actual things Drupal has done and this log is generated by code, not by an LLM and therefore will be completely accurate. However its what Drupal needs to know, it doesn't necessarily make sense to an end-user. We can at least improve it by making it so the labels are printed. But it will still be fairly techy.

  • πŸ‡©πŸ‡ͺGermany marcus_johansson

    First of all the Workspaces modules + WSE Config is really cool, however I did get stuck on most config entities where it currently won’t let me save them into a Workspace, which is how things are supposed to work as far as I could see. With Views it works.

    I could have the chatbot switch over to its own workspaces anytime it takes an action and currently I had to manually publish the Workflow But it’s some work to get something useful there both from our side and on, I’m guessing, the WSE’s side.

    In the Agent Interface we have added so that you can fetch all the content and config creations and changes as ids, this was not meant for the Chatbot UI from the beginning, but we have added a detail under the chat answer that can be toggled in the chatbot settings where you can see the actual changes made by the last response from the chatbot.

    What has also been added, so far in the taxonomy and fields agent, is that they will return links for every part of the system they change. This means that the Chatbot can be prompted to return this, which I have updated it to do on the Drupal CMS MR. So any beginner to Drupal would find links in the response where they can verify and see where changes are made.

    See screenshot below for example of both changes:

    Currently the agents are hardcoded not to do any deletions, the question is if we should also extend this to editing of configs that have substantial (10+) content items connected to them? This would make sure that you have a tool that can help you with the initial building, but that can’t really do any changes that are not revertable without a database backup/Workspaces?

  • πŸ‡¬πŸ‡§United Kingdom yautja_cetanu
  • πŸ‡¬πŸ‡§United Kingdom yautja_cetanu
  • πŸ‡¬πŸ‡§United Kingdom yautja_cetanu
  • πŸ‡¬πŸ‡§United Kingdom yautja_cetanu
  • πŸ‡¬πŸ‡§United Kingdom yautja_cetanu

    - We should have more details in the details. So if we're changing the name from X to Y it should say that. If its doing taxonomy terms it should show the term itself, not just the ID

  • πŸ‡©πŸ‡ͺGermany marcus_johansson

    From the fixed tickets and fixed structure on how to add original vs changed config and content.

    Diff on config entities:

    Labels on content entities:

    Later I will see if I can force this via interface, so anyone doing agents has to follow this. It would also help with fixing of revert of the last run agent(s)

  • πŸ‡¬πŸ‡§United Kingdom yautja_cetanu
  • πŸ‡¬πŸ‡§United Kingdom yautja_cetanu
Production build 0.71.5 2024