Add possibility to test with different prompts in the assistant

Created on 25 November 2024, 28 days ago

Problem/Motivation

Currently we are trying the same assistant prompts over and over, but we want to be able to try different prompts and see if we can get different results, depending on them, both in what actions are taken, but also to be able to adapt the resulting assistant message back to the user.

This means that we want to be able to both switch this via the Drupal admin system and see or filter it in the results.

The AI Assistant currently has some events and we could add a specific event for this. Currently these events does not allow to set prompt tags, but that could be added.

What we need in the end is:
1. A way to add a few new instructions and name them with some data name.
2. On the evaluations overview and detail page (if it exists), see and maybe filter on the datta name.
3. Be able via the UI to change which instruction is currently being used.

All should be done via the evaluations module, or if necessary with minimal changes to the assistants module.

Proposed resolution

One way is too hook into the AI pre-request event or the AI Assistant API events and change the prompt if the user has chosen an alternate instruction and add the data name to the prompt tags.

A simpler way is to actually change the configuration of the instructions on the AI Assistants, but via the UI element where you pick. And then just be aware of what is currently chosen.

Remaining tasks

User interface changes

API changes

Data model changes

📌 Task
Status

Active

Version

1.0

Component

Code

Created by

🇩🇪Germany marcus_johansson

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Production build 0.71.5 2024