Discussion - ECA actions, Chaining Augmentors

Some arguments against chaining different Augmentors:

1. There is probably not enough available resources to develop this. And this is my main argument and assumption.
2. Chaining will happen at API level at some point in the future. APIs will accept multimedia input (audio, image, document), and provide mixed media output. All kinds of data transformations will happen at API level.
3. Having in mind limited resources, focus could be towards creating powerful chat functionality which could allow to deal flexibly with multimedia input and output of APIs.

And what are arguments for?

Augmentors have been set up to handle a variety of outputs. When they return their data it is via a keyed array of arrays.

A simple example, from something like GPT3 for a summary:

{
"default": ["the response from the prompt"]
}

And when we ask for a list

{
"default": ["apple", "banana"]
}

Others are more complex, such as the Google Cloud Vision API - Safe Search, will return values with different keys. In this case the Augmentor will return something like...

{
"adult": ["UNLIKELY"],
"medical":["UNLIKELY"],
"spoofed": ["UNLIKELY"],
"violence": ["UNLIKELY"],
"racy": ["UNLIKELY"]
}

So, in conclusion we have come up with a flexible system to handle the expected outputs. If there is an API out there which returns different things, then the Augmentor implementation should be able to translate that across into the format.

This ticket description is about "chaining" Augmentors. Now that I have a better appreciation of ECA and how Actions/Activities/Tasks work, it is clearer that the orchestration of Augmentors is best handled inside ECA. Multiple Augmentors could then run in a pipeline.

For example,

On media save

1. If media type is a sound file
2. Run the Speech to Text Agumentor (on the file field, targetting the transcript field)
3. Run the Summarise Augmentor (on the transcript field, targetting the summary field)

The Augmentors have effectively been chained together.

Given the capabilities, I would say that ECA is the best way to handle this kind of functionality for "automation" of tasks.

If the editor is in the node edit screen and using the Field widget for Augmentors, they would need to press the right buttons in order to run these Augmentations in order. This would in essence be replicating the chained Augmentors.

It would be possible for us to define a "Next Agmentation" field on an Augmentor, however, that would be a poor replication of what was possible with ECA. ECA has the advantage of branching several actions from another, and so it would be more powerful that what we could do in a lightweight way.

The current approach we are taking here is to implement an Action interface to Augmentor so that it can take part in ECA pipelines.

Turning towards "chat" functionality... This is not trivial to implement as it involves semantic search and processing of the corpus or knowledge base for semantic meaning. Augmentors could play a role in that by summarising content, however, at some point in the pipleine you will need to use a semantic search API to vectorise your content. At this point in time this is out of scope for Augmentors. It would be best to implement some other integration, maybe with Search APi or perhaps a discrete sync and update function. This is the approach which has been taken with the QuickChat module. I can see this approach being taken for other semantic search backends. As I say, this is out of scope for what Augmentors is aiming to do just now.

We have been working on extending Augmentor to handle custom actions and specifically custom Actions which can be used by the ECA module. Our testing has also used the ECA VBO module to integrate with Views Bulk Operations as well.

Our work has been going well and there should be a release of Augmentor soon with the Actions included. ECA does allow for many actions to be defined and chained together. For example our testing has involved building Models which:
* Implement a VBO Action
* Trigger on a certain content type and field value
* Populate the body field with an Augmentor ection
* Populate other fields with other Augmentor Actions

So, for the purposes of this ticket, I reckon that we are close to achieving what we set out to do.

Does this mean that there is a chat system in place? No. However, this will allow for a lot of interesting and powerful implementations via the ECA module.

Once the Actions have been rleased, we will close this discussion ticket.

Good news.
So I will set aside time to test Augmentor, chaining augmentors and explore the gap needed for chat system.

Turning towards "chat" functionality... This is not trivial to implement as it involves semantic search and processing of the corpus or knowledge base for semantic meaning

Chat functionality is about gathering list of previous comments and sending them to openAI. Then chatbot writes a new comment - a response to the user(s).

Chat functionality is about gathering list of previous comments and sending them to openAI. Then chatbot writes a new comment - a response to the user(s).

True. The LLM gets the chat history gets the conversation as context and can answer based on that, however, it can only answer with knowledge encoded in the LLM. It won't know about the specific domain you are interested in. So if your Drupal website has domain specific knowledge (articles, FAQs, KB), it is unlikely that it will be returned. A chatbot should be able to return this kind of info to provide specific answers. This extra layer is what I was referring to.

he LLM gets the chat history gets the conversation as context and can answer based on that, however, it can only answer with knowledge encoded in the LLM.

An important point, and this will be one of the first ECA tasks I'm going to try and build once I get my hands on a working integration ECA - GPT integration. Because with ECA, we can load and use views results and I wonder if a prompt like "Based on the following knowledgebase articles: [view with KB nodes or even the full body texts], write a reply to [comment_text]" In other words, it depends on what the LLM knows, and how much we can instantly teach it. Also providing it a context will go a long way (like the Prompt module does, for instance). So we can define a context along the lines of "You are a virtual assistant only to help novice Drupal users solve common Drupal problems."

Ultimately, the ability for the AI of choice to remember prompts and responses by API key would be a great thing, however inevitably, one would use their own private AI with control over many important parameters.

Briefly speaking, The Augmentor AI module has now been updated to handle Actions. The module defines a new custom Action for each entity type. This is suitable for integration direct into sections of Drupal which deal with Actions. The module also implements a bunch of "minimal" Actions which are suitable for use in ECA. the minimal actions have a simpler Form such that it converts across well into the ECA Action template which is created.

As ECA is a flexible module, it is possible to chain together a number of different Actions and Conditions. It is a powerful system and has been tested to work with Augmentopr Actions.,

As this was a "discussion" ticket leading upt o the Actions implementation, I am going to close it out. Please download the latest version of Augmentor AI to give it a try. Thanks for everyone's contribuition here.

This is great news, thanks for making this possible. Would be great if you could add ECA to the "Ecosystem" section on the main page of this project, since it will then appear in the list of modules supporting ECA.

Cool. Will add ECA to ecosystem.

I'm trying to find out how to use the augmentor module in with ECA (and in general). Since there is no documentation anywhere yet, I hope I can ask a simple question that has me stuck.

What I'm trying to do is to construct a prompt that will do some SEO on commerce product entities. The prompt I'm looking to create (this is how I'm used to doing it with the Prompt module) is something like: "Write an SEO description based on the following product description: [commerce_product:field_product_description] and make sure to mention the name of the store: [commerce_product:store]. Limit the text to 160 words."

The behavior that I'm expecting, is that in the prompt field of my augmentor, I could enter this prompt and use tokens and then in ECA basically load the product and create an action that runs the augmentor holding the prompt, and define a target field (fi.e.: ield_seo_description).

Instead, what I'm getting are multiple source and target fields in the ECA action and an augmentor that just has {input} but no token browser.

I'm definitely on the wrong track in my approach to the module and would really appreciate some guidance to point me in the right direction, and will be happy to contribute that to any future manifestation of 📌 Start a User Guide for Augmentor Active

Thanks!