🇦🇺Australia @murrayw

The code has been working well in production for us for quite some time. It is possible that that a success message may be shown if there is an error or a failure of some kind.

When we make the request we should be getting no exceptions, no timeout, a response, a 200, a payload, a success. Eleo, can you please check that we have robust checking in place for the sync.

fubarhouse → credited murrayw → .

Hi ressa, as mindaugasd says, the Augmentor module provides a plugin system for integrating external AI services. If you would like to integrate in Hugging Face a new Contrib module would be needed which implemented an Augmentor plugin. An augmentor provides the ability to customise the prompt and to send it through to an external service which returns some sort of payload. The payload is then put into a response (associative array) on an endpoint for consumption by Drupal (node edit, or some other code). The big question is which API (method) you would like to use. Can you provide a link to documentation in Hugging Face to the API you would like to call. We could then make some recommendations as to feasibility. At the end of the day, a new contrib module would be needed to implement it.

I can see that you are indeed getting results back as JSON. This is good. The next step is to make sure that they are being transformed properly. We use a module called JSON Template to manage that on the client side. You will need to configure the block to use a template to transform the results. Please make sure that the block has been configured to use a template.

The Recombee module ships with a simple Handlebars template which will print out the title for the items coming back. Please take a look in the module to see how this is working. If you wish to customise the the response, you will need to implement your own handlebars template, using thsi initial one as a starting point. The handlebars template can be placed either into a module or into the theme as it is discoverable there as well.

This sounds like a great idea for a submodule in Augmentor. As a suggestion: augementor__search_api_processors

VladimirAus → credited murrayw → .

VladimirAus → credited murrayw → .

I will make a brief comment here because @mindaugasd has alerted me to this thread

It sounds like Prompt AI is doing a lot of things which Augmentor is also doing. Basically integrating with a variety of services and exposing the resulting data back into Drupal to augment content in a variety of ways. Augmentor now integrates with ECA and I have recently presented on this at DrupalSouth 2023 (Wellington). The video should be available soon to see how that is all working. So yeah - these two modules appear to be doing similar things.

Eleo, do you have comments on this code suggestion?

murrayw → created an issue.

Cool. Will add ECA to ecosystem.

Hi fishtree. The tutorial looks good. It has a lot of prompts which could easily be wrapped up as Augmentors and exposed in the WYSIWYG or as field widgets. There are a lot of good ideas in there.

In regards to the structure data... if I get the gist of it I believe this is the prompt you are referring to:

Can you provide a report on the paragraph? The report should include document statistics, vocabulary statistics, readability score, tone type (available options are Formal, Informal, Optimistic, Worried, Friendly, Curious, Assertive, Encouraging, Surprised, or Cooperative), intent type (available options are Inform, Describe, Convince, or Tell A Story), audience type (available options are General, Knowledgeable, or Expert), style type (available options are Formal or Informal), emotion type (available options are Mild or Strong), and domain type (available options are General, Academic, Business, Technical, Creative, or Casual).

ChatGPT retiurns data back as text - list items. From what I can see this is not JSON, although, i may be mistaken. Aslo, does ChatGPT always return data in exactly the same format.

My hunch is that the text feedback is not going to be structured enough. It won't be structured wnough for general cases at least. If you believe that it is possible, then you could implement a specialised ChatGPT augmentor to return the data. It could return it as a JSON "string". The challenge then is working out how to handle JSON. Currently Augmentors pretty much assume that the response is a string and most of the integrations are set up around that.

Now, Augmentors do returned a keyed array, and so, they do support some structure - pretrty much what you are discussing.

I would recommend implementing your own Augmentor which is apple to translate the results to something which is more structured.

Unfortunately, at this stage it is not possible to target the Summary field when it is part of the body. This is a limitation of the module as tehre is no way to target the field properties in this way.

In order to achieve what you want, the Summary would need to be split from the Body field so that it could be targetted individually.

For the future this could be a feature request. Adding it in would require a fair bit of effort as field widgets and Actions would need to be updated in order to support it. We do not want to support this at this time.

As a workaround, you could define your own field widget that specifically targetted the summary field. So, it would be possible to take what is in the module and adapt it to your own purposes.

Briefly speaking, The Augmentor AI module has now been updated to handle Actions. The module defines a new custom Action for each entity type. This is suitable for integration direct into sections of Drupal which deal with Actions. The module also implements a bunch of "minimal" Actions which are suitable for use in ECA. the minimal actions have a simpler Form such that it converts across well into the ECA Action template which is created.

As ECA is a flexible module, it is possible to chain together a number of different Actions and Conditions. It is a powerful system and has been tested to work with Augmentopr Actions.,

As this was a "discussion" ticket leading upt o the Actions implementation, I am going to close it out. Please download the latest version of Augmentor AI to give it a try. Thanks for everyone's contribuition here.

The Augmentor AI module has now been updated to handle Actions. The module defines a new custom Action for each entity type. This is suitable for integration direct into sections of Drupal which deal with Actions. The module also implements a bunch of "minimal" Actions which are suitable for use in ECA. the minimal actions have a simpler Form such that it converts across well into the ECA Action template which is created.

So bottom line, If you wish to automate nodes in ECA use the Content item - minimal Augmentor. This will work well in ECA.

Now that we have Actions, we have a way to automate them with ECA. And extended on from that, the ECA VBO module will expose an action event for Views VBO. So with this module you can bulk process nodes with an Augmentor via complex automations through ECA.

On the ChatGPT side of the equation we have the ChatGPT Augmentor Module
https://www.drupal.org/project/augmentor_chatgpt →

It does what is days on the box - bring Chat GPT cunctionality to the Augmentor ecosystem.

Finally, picking up on ALfTheCat's comment above - yes, that is now possible.

As this was a "discussion" ticket leading upt o the Actions implementation, I am going to close it out. Please download the latest version of Augmentor AI to give it a try. Thanks for everyone's contribuition here.

Chat functionality is about gathering list of previous comments and sending them to openAI. Then chatbot writes a new comment - a response to the user(s).

True. The LLM gets the chat history gets the conversation as context and can answer based on that, however, it can only answer with knowledge encoded in the LLM. It won't know about the specific domain you are interested in. So if your Drupal website has domain specific knowledge (articles, FAQs, KB), it is unlikely that it will be returned. A chatbot should be able to return this kind of info to provide specific answers. This extra layer is what I was referring to.

Thanks for the code WalterP. Eleo passing this over to you to see.

We have been working on extending Augmentor to handle custom actions and specifically custom Actions which can be used by the ECA module. Our testing has also used the ECA VBO module to integrate with Views Bulk Operations as well.

Our work has been going well and there should be a release of Augmentor soon with the Actions included. ECA does allow for many actions to be defined and chained together. For example our testing has involved building Models which:
* Implement a VBO Action
* Trigger on a certain content type and field value
* Populate the body field with an Augmentor ection
* Populate other fields with other Augmentor Actions

So, for the purposes of this ticket, I reckon that we are close to achieving what we set out to do.

Does this mean that there is a chat system in place? No. However, this will allow for a lot of interesting and powerful implementations via the ECA module.

Once the Actions have been rleased, we will close this discussion ticket.

The recipe is as follows:
* In the prompt for GPT3 (or whatever) ask for it comma separated.
* In the field widget, make sure the "Explode separator" is set to ","

Prompt

Extract keywords from this text and comma separate them:

{input}

See attached screenshot for how the field widget has been done.

Augmentors can return an array of options for a key, eg "default". This structure was decided on so that different options could be displayed to the user as a convenience. The UI will display the options if they are available. I do not believe it to be a bug.

I will hand over to Eleo who can provide more insight and expand on this answer.

There is little to go on in this report. More information will be required.

* A related issue was linked to. Will the work done in that issue resolve the one here?
* Which Augmentor were you attempting to integrate in this issue? Was it something different to ChatGPT Augmentor?
* Do you have any further details on this issue or other related information?
* Have you done any debugging which would help identify the issue?

Augmentors have been set up to handle a variety of outputs. When they return their data it is via a keyed array of arrays.

A simple example, from something like GPT3 for a summary:

{
"default": ["the response from the prompt"]
}

And when we ask for a list

{
"default": ["apple", "banana"]
}

Others are more complex, such as the Google Cloud Vision API - Safe Search, will return values with different keys. In this case the Augmentor will return something like...

{
"adult": ["UNLIKELY"],
"medical":["UNLIKELY"],
"spoofed": ["UNLIKELY"],
"violence": ["UNLIKELY"],
"racy": ["UNLIKELY"]
}

So, in conclusion we have come up with a flexible system to handle the expected outputs. If there is an API out there which returns different things, then the Augmentor implementation should be able to translate that across into the format.

This ticket description is about "chaining" Augmentors. Now that I have a better appreciation of ECA and how Actions/Activities/Tasks work, it is clearer that the orchestration of Augmentors is best handled inside ECA. Multiple Augmentors could then run in a pipeline.

For example,

On media save

1. If media type is a sound file
2. Run the Speech to Text Agumentor (on the file field, targetting the transcript field)
3. Run the Summarise Augmentor (on the transcript field, targetting the summary field)

The Augmentors have effectively been chained together.

Given the capabilities, I would say that ECA is the best way to handle this kind of functionality for "automation" of tasks.

If the editor is in the node edit screen and using the Field widget for Augmentors, they would need to press the right buttons in order to run these Augmentations in order. This would in essence be replicating the chained Augmentors.

It would be possible for us to define a "Next Agmentation" field on an Augmentor, however, that would be a poor replication of what was possible with ECA. ECA has the advantage of branching several actions from another, and so it would be more powerful that what we could do in a lightweight way.

The current approach we are taking here is to implement an Action interface to Augmentor so that it can take part in ECA pipelines.

Turning towards "chat" functionality... This is not trivial to implement as it involves semantic search and processing of the corpus or knowledge base for semantic meaning. Augmentors could play a role in that by summarising content, however, at some point in the pipleine you will need to use a semantic search API to vectorise your content. At this point in time this is out of scope for Augmentors. It would be best to implement some other integration, maybe with Search APi or perhaps a discrete sync and update function. This is the approach which has been taken with the QuickChat module. I can see this approach being taken for other semantic search backends. As I say, this is out of scope for what Augmentors is aiming to do just now.

The field widget only handles the targetting of direct fields on the entity. It may be possible to add the field widget into the Paragraphs form and have it work that way, so it may be possible,

Assigning this across to Eleo to comment on whether it is possible and what it would take to achieve this.

We are currently scoping out what Actions might look like and how they can be implemented in a flexible way so that ECA and others can be handled. The aim will be to be able to use all types of Augmentors in the Action so this is not limited to ChatGPT, GPT4 etc.

The ability to chain Augmentors is a powerrful concept. The orchestration of this could be done in ECA or Rules, however, I would need to look into those systems to see how they can be chained together.

I will keep this issue open for now for discussion and any futher ideas which there may be out there.

We will create a new Feature ticket for Actions once our ideas have settled down a bit.

There is a link at the bottom of the project page:
"Join #augmentor and discuss the future of Augmentor with us."

Back again with a few more thoughts about building a chat bot and extending on from the knowledge base comments for Qucikchat AI...

It is important to note that LLMs come pretrained. It is is possible to finetune and prompt engineer to get certain results. However, in order to get the most from a knowledge base it is required to use embeddings to understand the content and what is being asked of it. The basic ideas is that

• Your knowledge base is analysed up front for semantic meaning.
• Then the question asked by the user undergoes a similar process so that the most relevant pieces of information can be extracted from the knowledge base.
• Once these parts have been found, they are sumbitted to the LLM with a prompt to get a conversational response.

See the following helpful articles on how this can be achieved:

• https://www.mlq.ai/fine-tuning-gpt-3-question-answer-bot/
• https://betterprogramming.pub/how-to-give-your-chatbot-the-power-of-neur...
• https://www.lennysnewsletter.com/p/i-built-a-lenny-chatbot-using-gpt

So, in a nutshell, the content in Drupal will need to be analysed first, so that the chatbot can be customised to the content on the site in question. Quickchat AI makes this process easier because it handles the embeddings, the prompt and finally the widget to make it all work. The Quicchat AI module is really just about submitting the knowledge base to Quickchat which then does the processing work. ie. Drupal is used as the source of truth for the content and the functionality and processing is offloaded to an external service.

Of course, this functionality could be implemented in Drupal, The above tutorials show how that could be done. It would however require a an intermediate step before ECA came into the picture. Building and maintaining a database of embeddings for site content is a potential way forward for someone wanting to support this funtionality.

The future - it is easy to see that OpenAI (and other AI providers) will start injesting sites automatically and create embeddings for them. Search engines will not just be for keywords, but for semantic emeddings of meaning. The chatbot functionality could then be easily added to the site. This will probably happen very quickly and so, the effort to manage this in Drupal may be wasted.

From my point of view, using Drupal to maintain a knowledge base which can be used as a source for services such as Quickchat AI is a sensible way forward as it leverages Drupal for what it is good at.

Finally, this doesn't leave ECA out of the picture. Processing nodes and other entities via ECA and Augmentor is a natrural approach to take to bring AI to your content.

It is also worth adding some background to Quickchat AI module, as there are some relevant insights from that project which will be relevant to future AI integrations in Drupal.

Qucikchat AI is a paid service which prevides chatbot functionality on top of OpenAI. One fo the interesting things about Quickchat is that it provides an API for syncing across a "knowledge base" to Quickchat Ai which is used in defining the custom content for the chatbot. ie. The content in Drupal, is driving the experience. When a user submits a question to the chatbot, the knowledge base can be referred to before getting the response from OpenAI.

This kind of model - Drupal as the content source, will no doubt be a pattern which is repeated in the future as deeper AI integrations are made which use the CMS as the source of the content.

Thanks for the review of the modules. I am responding as one of the maintainers of Augmentor. It is nice to see that it is recognised as being flexible and with good code quality. I'd like to provide some more background on Augmentor.

Firstly, the module was created out of a recognition that many AI solutions in Druapl (2019 to 2022) were bespoke and tended to solve the same issues over and over. eg. The development of widgets in the node edit form to expose the AI functionality. So Augmentor aimed to provide an implementation for these UI features so that future AI modules did not need to reinvent the wheel on this.

Secondly, the module was created to provide a pluggable system for AI integrations. A review was done of the common patterns for how AI modules worked (config, input, keyed array of outputs) - the concept of an "Augmentor" was based on this. So there is a key abstraction of what an AI integration would look like.

Finally, we built out an ecosystem of important integrations: Open AI (GPT3 completion), Google Text to Speech, Google Speech to Text, Google Cloud Vision. This kicked off the ecosystem. Recently we have added OpenAI ChatGPT and NLP Cloud. Adding a new integration is pretty straight forward as all the developer needs to handle is the connection and the config for how the integration is to work.

We are encouraging the development of a vibrant ecosystem around the module. It would be great to see the addition of new sub modules supporting new AI systems. We are also working on extending the intergration points into Drupal. If you are considering working on AI in Drupal, I would encourage you to take a look at the Augmentor AI module and its ecosystem.
https://www.drupal.org/project/augmentor/ecosystem →

The votes from the Sydney meetup are in are in:
Option 1: 8 - Layouts nested under appearance
Option 2: 2 - Layouts nested under the theme
Option 3: 1 - No change = keep it in structure

So option 1 was the winner.

The tradeoff was between depth and encapsulation. In general people thought it would be quicker to get to if it was at a lower depth. The two votes for having it in the theme considered encapsulation to be a sronger organising principle. The one vote for no change was from a developer who didn't want to change his ways. In general moving it to apopearance and out of structure was considered a good thing.

Thanks @pameeela @skipper-vp @marji @andrekakos @jct321 @jimik42 @richo_au There were a number of others there but I was unable to find user accounts for them. My apologies if I missed you out in this list.

Hi Vit, is this something you could take a look at? If not, just let me know.