[META] Drupal could be great for building AI tools (like ChatGPT)

Drupal AI news - 2023 March 10

The best modules in different categories

• Module with the best name* thus far: AI https://www.drupal.org/project/ai → . Module was created just few days ago. Name possibly could be further improved to "Artificial intelligence" or "AI framework". *opinion.
• Module which has most vibrant community and development: OpenAI / ChatGPT / AI Search Integration →
• Module which has most flexibility and quality: Augmentor →

Module first to implement CHAT with ChatGPT API

No one yet.

Recent key articles in Drupal AI world

• Artificial Intelligence, the future of Content Management and the Web by Dries Buytaert (2023-02-27)
• [video] OpenAI ChatGPT Comes to Drupal by OSTraining (2023-02-23)

Related issues:

• 📌 Use ChatGPT for solving Drupal issues to increase rate of development Active

Communication channels:

• Come join central slack channel #ai
• There are other channels (and more active) for different modules which can be found on module project pages.

I asked GPT-4 to generate user story examples for development of Drupal AI. I encourage everyone to chat about Drupal with AI.

Also explored creating an AI module using GPT-4 itself. Published case study here: #3336313-6: Use ChatGPT for solving Drupal issues to increase rate of development →

1. As a content creator, I want the AI to generate article drafts based on given keywords or topics, so that I can save time in the content creation process and focus on refining the generated drafts.

2. As a website owner, I want to provide personalized content recommendations to my users based on their browsing history and preferences, so that I can enhance user engagement and increase the time users spend on my website.

3. As an e-commerce website administrator, I want an AI-powered chatbot that can intelligently guide users through the product catalog and answer frequently asked questions, so that users can find the information they need quickly and the support team can focus on more complex queries.

4. As a site builder, I want a collection of pre-built AI-powered components, such as blocks and widgets, that I can easily drag and drop into the site design, so that I can add AI functionality to the site without extensive coding.

5. As a multilingual website owner, I want to use AI to automatically translate my site's content into multiple languages, so that I can reach a broader audience and cater to users with diverse language preferences.

6. As a community manager, I want an AI-powered content moderation system that automatically removes any spam or offensive content from my site, so that the online environment remains safe and welcoming for all users.

7. As a marketing manager, I want AI-driven sentiment analysis of user comments and reviews on my website, so I can better understand user opinions and preferences and adjust our marketing strategies accordingly.

8. As a developer, I want access to a well-documented AI Framework API, so I can build custom AI-powered features and extend the core functionality of the Drupal AI module easily.

Thanks for the review of the modules. I am responding as one of the maintainers of Augmentor. It is nice to see that it is recognised as being flexible and with good code quality. I'd like to provide some more background on Augmentor.

Firstly, the module was created out of a recognition that many AI solutions in Druapl (2019 to 2022) were bespoke and tended to solve the same issues over and over. eg. The development of widgets in the node edit form to expose the AI functionality. So Augmentor aimed to provide an implementation for these UI features so that future AI modules did not need to reinvent the wheel on this.

Secondly, the module was created to provide a pluggable system for AI integrations. A review was done of the common patterns for how AI modules worked (config, input, keyed array of outputs) - the concept of an "Augmentor" was based on this. So there is a key abstraction of what an AI integration would look like.

Finally, we built out an ecosystem of important integrations: Open AI (GPT3 completion), Google Text to Speech, Google Speech to Text, Google Cloud Vision. This kicked off the ecosystem. Recently we have added OpenAI ChatGPT and NLP Cloud. Adding a new integration is pretty straight forward as all the developer needs to handle is the connection and the config for how the integration is to work.

We are encouraging the development of a vibrant ecosystem around the module. It would be great to see the addition of new sub modules supporting new AI systems. We are also working on extending the intergration points into Drupal. If you are considering working on AI in Drupal, I would encourage you to take a look at the Augmentor AI module and its ecosystem.
https://www.drupal.org/project/augmentor/ecosystem →

It is also worth adding some background to Quickchat AI module, as there are some relevant insights from that project which will be relevant to future AI integrations in Drupal.

Qucikchat AI is a paid service which prevides chatbot functionality on top of OpenAI. One fo the interesting things about Quickchat is that it provides an API for syncing across a "knowledge base" to Quickchat Ai which is used in defining the custom content for the chatbot. ie. The content in Drupal, is driving the experience. When a user submits a question to the chatbot, the knowledge base can be referred to before getting the response from OpenAI.

This kind of model - Drupal as the content source, will no doubt be a pattern which is repeated in the future as deeper AI integrations are made which use the CMS as the source of the content.

Thanks for creating Augmentor ecosystem. Maybe more marketing will be needed (video maybe?)
Personally, I already built a flexible AI system with custom code many months ago, so I have limited incentive to use contrib modules myself, but will try to test/help to some degree.

Interesting point:

Drupal as the content source, will no doubt be a pattern which is repeated in the future as deeper AI integrations are made which use the CMS as the source of the content.

There is an issue of DrupalGPT 📌 Training a Large Language Model for Code (Code LLM) - an expert Drupal A.I. Active to create a robot with very high degree of Drupal expertise, which could help develop Drupal itself.

But also making an AI aware and use content of any Drupal website in an interesting angle.
Beyond Drupal, everyone across the world will need this functionality to train A.I. in all kinds of knowledge ecosystems. I wonder what the accessible and thorough solution to this will be.

Back again with a few more thoughts about building a chat bot and extending on from the knowledge base comments for Qucikchat AI...

It is important to note that LLMs come pretrained. It is is possible to finetune and prompt engineer to get certain results. However, in order to get the most from a knowledge base it is required to use embeddings to understand the content and what is being asked of it. The basic ideas is that

• Your knowledge base is analysed up front for semantic meaning.
• Then the question asked by the user undergoes a similar process so that the most relevant pieces of information can be extracted from the knowledge base.
• Once these parts have been found, they are sumbitted to the LLM with a prompt to get a conversational response.

See the following helpful articles on how this can be achieved:

• https://www.mlq.ai/fine-tuning-gpt-3-question-answer-bot/
• https://betterprogramming.pub/how-to-give-your-chatbot-the-power-of-neur...
• https://www.lennysnewsletter.com/p/i-built-a-lenny-chatbot-using-gpt

So, in a nutshell, the content in Drupal will need to be analysed first, so that the chatbot can be customised to the content on the site in question. Quickchat AI makes this process easier because it handles the embeddings, the prompt and finally the widget to make it all work. The Quicchat AI module is really just about submitting the knowledge base to Quickchat which then does the processing work. ie. Drupal is used as the source of truth for the content and the functionality and processing is offloaded to an external service.

Of course, this functionality could be implemented in Drupal, The above tutorials show how that could be done. It would however require a an intermediate step before ECA came into the picture. Building and maintaining a database of embeddings for site content is a potential way forward for someone wanting to support this funtionality.

The future - it is easy to see that OpenAI (and other AI providers) will start injesting sites automatically and create embeddings for them. Search engines will not just be for keywords, but for semantic emeddings of meaning. The chatbot functionality could then be easily added to the site. This will probably happen very quickly and so, the effort to manage this in Drupal may be wasted.

From my point of view, using Drupal to maintain a knowledge base which can be used as a source for services such as Quickchat AI is a sensible way forward as it leverages Drupal for what it is good at.

Finally, this doesn't leave ECA out of the picture. Processing nodes and other entities via ECA and Augmentor is a natrural approach to take to bring AI to your content.

Hi,

I have some thoughts related to AI and Drupal. Mostly services with Drupal are traditional content management related.
But what if Drupal site is a learning environment, or a social media platform? Then there might be different needs for AI integration.
Here are some what came into my mind as a "hobby" creator of a social learning environment with D9.

1. Simple chat with AI for each users (students). Similar user experience is enough as now the openai website provides. I guess this is not yet done.
2. Learning content creation. There are some learning related modules like Quiz and H5P, where a content creator could ask for AI to create a learning content from the topic X and ECA for example would insert that in the content type. For example create a multiple choice question set about topic X. Tried it, works.
3. Grading automation for student created essays. For example if students are required to post an essay about topic X. Could AI create "pre grading" and grade the students submitted essays (nodes) automatically using a certain grading criterias? Now teacher needs to manually read all the essays and grade them, it is slow.

Then another thing is related to social media platform and content moderation. Could AI block users automatically who posts nude pictures of hate speech? Or some ECA action.. I guess with AI all this could be possible, but not yet a working solution with Drupal? And it might be expensive also if loads of content to analyze.
Then the last idea is related to user profiles, when there are loads of data like hobbies, interests or expertices. Could AI act like a personal coach, and align its output related to users profile? Also if a social media platform asks users mood, there could be analysis and automated "cheer up" responses from the AI.

I just heard about the new Open Source AI project Open Assistant.

• Open Assistant - chat based large language model (Github)
• A ChatGPT Alternative That's Free & Open Source!

It is still early, but the fact that it's free and Open Source is a big plus in my book.

• Added info about #ai slack channel to the description.
• Updated/simplified list of Drupal AI modules (still keeping it in the comment, because of frequent updates)

Is this possible with ChatGPT to have a content recommendation? How can this be achieved?

@pinkonomy I explored this successfully with movies.
You give AI a few movie examples, and GPT-4 recommends related movies. It does this with very high quality and explains the reasoning why something is a good choice.

So to create recommendation you give GPT:
1) previous examples what user liked
2) give list of options what could be recommended (content or titles from your site)
3) ask GPT to respond with recommendation in particular format (json maybe)

This can be done on a drupal side with prompt engineering tools. The beginning is here https://www.drupal.org/project/prompt → and https://www.drupal.org/project/search_api_ai → . Also I applied in Drupal innovation contest for funding 🌱 Fully fledged website and module development with Artificial Intelligence Active to improve some part of prompt engineering toolbox (results will be on Monday if selected).

Another part what is needed is larger context window to provide GPT with more information. I wrote about increasing of context window in this issue: 🌱 Fully fledged website and module development with Artificial Intelligence Active (summary: we need to wait until larger context window is accessible and affordable, but solutions can programmed today)

Final part is caching and displaying recommendation results to the user. (this part unclear if modules exist. Maybe someone else can provide more info here)

This is the simplest method and which will probably be most popular for small-average size sites. For large sites, solutions might be a bit more sophisticated.

This initiative participated in Drupal innovation contest → . I tried to shape this project to cover most important aspects of AI and increase development resources by using AI itself.

My expectation was that half of other pitches will be about AI.
I thought - I will fill some stars (displayed in the pitch), and other winners will fill the other.
Sadly, there was only 1 AI pitch which itself lost. This of course is sad for me. I would like to learn some critique - what went wrong?

I hope there was something happening at the conference about AI development behind the scenes.

There is also possibility of funding still.

On the other hand, I don't know why almost nobody is interested in Chat UI ✨ ChatGPT-like UI/UX (and integration with aichat) Needs work for example (6 subscribers), I think this is a cool thing. I would say that issue is 5% done of what is possible. [Also, this is an opportunity for Drupal, but other projects will take the market share of this particular feature]

Also - I wonder how much new people Drupal attracts. Drupal was amazingly good 15-10 years ago. So much innovation was created in the short time. Right now, if I would be young, following AI developments, I would go on some other path. In other words, Drupal is probably not attracting people who want to build the next big thing. There is a lot of talk about ambitious site builders, but maybe Drupal is mostly at senior level now and still does not come back to its roots (yet?)

Maybe it was too theoretical?

I think you need to get those who haven't seen what it's capable of excited. If I were you, I would record a ChatGPT session, where you build a small Drupal module in ten minutes, which would normally take 1-2 hours.

Also, I think Drupal should lead with the choices, and focus on cooperating with AI Open Source projects such as Open Assistant and Hugging Face Chat UI.

I think it would be cool to cooperate with these projects, and build a prototype feeding their LLM's drupal.stackexchange.com, drupal.org, API.Drupal.org, etc. and iterate to build a cool and useful Drupal tool, on open source foundation, eventually better and freer than ChatGPT.

Thanks @ressa, it is always uplifting to see some other enthusiastic person.

I did build Drupal module in 5 hours, which would have taken 40 otherwise. And with a lot better quality. But chose not to show it, because I think audience would not have liked that. Because it can be seen as AI replacing jobs and I don't want pitch to be scary. I talked about adapting instead.

I decided to display something visual like changing theme colors, considering there is only 2.5 minutes (by the way, for some reason pitch was reduced to 2 minutes during conference by decision or accident, which was not good)

About open LLMs. Personally, I focus on practical application - it means creating Drupal features. It is a priority for me. Commercial models achieve that in my case. If I would do LLM, then features would be undone.

@mindaugasd
One question,did you use ChatGPT-4 for the 5 hours module creation or something else?

@pinkonomy I use custom primitive agents with GPT-4 model API. It is displayed in video, but I simplified it for video.
Another difference - I copy the code for now (while video shows automatic code editing prototype).
Also check out this example from 3 months ago about module creation in detail: #3336313-6: Use ChatGPT for solving Drupal issues to increase rate of development →

Maybe people don’t want their jobs eliminated by robots.

@cilefen that's the thing. My video is exactly a proposed solution to this problem: Adapting.
But there is another thing, not only jobs, but expertise. If someone prides themselves being an expert, then chatGPT takes that away.

@mindaugasd: You're welcome! I think we shouldn't fear this new invention, but look at it with an open mind, and see the dangers and down sides, as well as opportunities.

Interesting info about the module you built, and your 1:8 ratio in terms of saved time (5/40 hours) aligns with my estimate (10:60-120 minutes).

About replacing jobs, I get it. It's scary, but the taking over of tasks is going to happen. It's like horse carriage chauffeurs looking at cars in 1905, or table-tr-td-superheroes looking at CSS replacing tables twenty years ago: Why should we use that?! Or me, looking at Composer taking over Drupal ten years ago: "What's wrong with the tarball?" But things change.

As you write, we look at it from two different perspectives: I can see an almost immediate pay off using AI as a ressource for learning and building Drupal by training an LLM on Drupal ressources, and at the same time an opportunity to guide the development in an Open Source direction, for a less monopolistic, proprietary, black box world of Google, Facebook, Microsoft, Amazon, etc.

You are more into content creation. Both are very relevant. If someone sponsored you for working on a Drupal LLM ressource with Open Assistant/HuggingChat, would you be open for that?

@ressa If funding would be on the table for open LLM, possibly I would refuse or agree after sorting out a lot of doubts.

Benefits of open LLMs and open source software cannot be directly compared. AI is very powerful tech. In the wrong hands without safety rules it can cause a lot of trouble. While OpenAI was created to make AI safe, many open LLMs are create to remove safety features. Overall AI safety is super complicated subject. The biggest and hardest problem to solve ever, so I don't have clear answers here.

About replacing people - it is a real issue too. It does not effect me personally, but I understand it can be devastating for people.

But AI is a natural progression of civilization, there is no alternative. It was always destined to happen sooner or later, and personally I want to participate in stearing a ship a little little bit instead of ignoring the problem does not exist.

What is better direction - to be prepared for what is coming or unprepared? I choose option A.

Created a new issue to explore solving things we talked about "Learning about artificial intelligence and discussing its role within Drupal and Drupal community": #3365988: Learning about artificial intelligence and discussing its role within Drupal and Drupal community →

Drupal AI meetup

We will be having online meetup of Drupal AI community.
More information in this issue: #3365988-6: Learning about artificial intelligence and discussing its role within Drupal and Drupal community →
Further meetups or updates will be announced in that issue as well.

Created a new issue to explore business side of AI building:
#3372329: Business case for AI building →

Created a new issue for studying AI safety
#3372687: Studying AI safety →

Significant milestone for Drupal AI is achieved today - AI prompt engineering → module is released. It lets build prompts combining all kinds of Drupal data, it also acts as a database of prompts. For ultra flexibility - module has both "content" and "configuration" entities - letting AI builder decide what is best for their usecase.

On the other hand. module is a building block which has to be integrated with other AI modules to be valuable. Since module is new, there are no integrations at the moment.

Also I released or initiated many other AI modules in August. Take a look at AI initiative → page to learn more.

Also see full and constantly updated list of all Drupal AI modules at the top of this issue. 📌 [META] Drupal could be great for building AI tools (like ChatGPT) Active

Drupal has new great new module ecosystem for building of AI tools https://www.drupal.org/project/ai_interpolator →
This website https://workflows-of-ai.com/ have many videos showcasing how to use it and build AI features in the easiest most powerful way thus far. There is also an introductory video in depth: https://www.youtube.com/watch?v=Sw1Zk__M16c

List of Drupal AI modules from 1st comment I moved to separate issue so that updates to the list could be tracked: #3429343: Overview of Drupal AI modules in one place →

@mindaugasd are you able to move this one over to the AI initiative project? I don't seem to be able to.

Done