AI sub module :ECA support (ai_eca)

ok I've made a port of the ECA module.
Some remarks.

1. @marcus: I havent done the STT and TTS code yet so I'm gonna need some help there.
2. @kevinquillen: I'm not knowledgable on ECA and how the actions should work, I dont see errors so I presume they're working. Could you test the Chat / Completion / Moderation action?

Thanks!

Added the STT and TTS code as per the examples, but I have not tested this implementation.
Someone with more ECA experience should probably look at this.

A quick look at the code and I saw thtat this will need to be refactored to handle all providers and be configurable per ECA rule, currently its hardcoded to default provider or OpenAI.

I might have time to look into refactoring it later, but I have very little time at the moment.

No Marcus,

I'm planning to use your model selector to let the user choose the Provider/model per rule.
Let me work on it some more.

Thanks lammensjs for adding tests!

Some issues that we should fix - I wrote to Jürgen and he confirmed that we have to have a free text textarea configuration for the model configuration, let me know if you need help with that:

• The chat label can't be named ChatGPT
• Temperature and max_tokens has to be moved into a large configure textarea since not all providers/models have those.
• System prompt should be empty by default and not sent when empty since certain models do not allow it.
• System prompts role have to be renamable, since it has to be assistant on certain models.
• We need validation on the form (though that could potentially wait for version 2)
• We should remove completion for now since we don't have that operation type.
• Embed needs configuration text area.
• The moderation label shouldn't have OpenAI in its name.
• Moderation should have a drop down to choose as well.
• Language, Task, Voice and Response Format in Speech to Text should be in a configuration textarea, those are OpenAI specific.
• The model in Speech to Text is linking to OpenAI.
• Text to Speech has similar problems.

#10 this is correct.

Note that ECA 2 also has the ability to write to files from base64 responses, which should enable creating new actions that can generate images or binary files from AI.

For the influxdb → -module I used a textarea and the core Yaml-processor to ensure validness and correctness. See https://git.drupalcode.org/project/influxdb/-/blob/2.0.x/modules/bucket_...

Would that be something that could work here as well?

lammensj → changed the visibility of the branch 3462735-ai-eca2 to hidden.

Short recap of yesterdays work:

• Removed the Completion-action
• Moved the model-config item to the base-class ("AiActionBase")
• Added kernel tests for Embedding and Moderation
• Added separate textarea for model-specific config to new base class "AiConfigActionBase". This one is validated via the YamlParser-service of ECA and via the Symfony Validator. Based on provider API defaults (eg. https://git.drupalcode.org/issue/ai-3462735/-/blob/3462735-eca-config/mo...), I can create constraints and use those to validate the content of textarea.

Things yet to do:

• Add kernel-tests for TTS and STT
• Ensure that violations of the model-specific config are readable by the user

ok so I also did some things:

• removed all occurences
• Moved into textarea config
• . removed and added in description
• made dynamic via configration form
• We need validation on the form (though that could potentially wait for version 2)
• Removed
• Added
• Removed
• Check
• Added this and examples in textareas
• removed reference
• applied all here too.

The configuration textarea I added looks like this:

@Lammensj : I think you can replace this by your global yaml thing.
I have tested the STT / TTS with these settings and it does work.

@Marcus: you say: Embed needs configuration text area
It's not in the call to the embedding model. I've added the possibility to add config, but if you could give some examples then I will add them in the form as example.

@lammensj we're working in 3462735-eca-config right?
Will you remove the other branch? Or do I do it?

@wouters_f - actually Bedrock provider would be the best example of this when its there, but you can look at OpenAI embeddings, they should have dimensions that are not hard coded.

So for instance this config should matter there at least:
dimensions: 1024

But most of them do not allow that to be touched, so I think that is the only provider at the moment.

lammensj → changed the visibility of the branch 3462735-eca-config to hidden.

@wouters_f, I've merged all the work in the original branch you created "3462735-ai-sub-module", together with the latest changes. I've also added a new example flow, based on the flow you created but with the adjusted configuration per action. Would you be able to test it? I don't have access to an OpenAI API key, only Ollama via HuggingFace but that doesn't have support for all operations.

One thing I noticed: the STT-action originally had a "task" config option, for which you could choose between "transcribe" and "translate". Currently, only the OpenAi-provider has support for this operation, but the transcribe-method is hardcoded. @wouters_f, @Marcus_Johansson, any preference on how to deal with this?

This is getting merged before the next alpha. I will set it as experimental for now. Thanks everyone!

Automatically closed - issue fixed for 2 weeks with no activity.