🇱🇹Lithuania @mindaugasd

mindaugasd → created an issue.

mindaugasd → created an issue.

Related issue where this was discussed in the past: ✨ Add support for language translation, and non-English languages Active

Renamed "interpolator" with "automator" in description.

Mostly I am looking forward to use this feature with AI → module automator's. In the meantime will do it programatically: db query, a batch and AI module's abstraction API.

In my case filling a single field on many nodes (no chaining).

mindaugasd → created an issue.

Looks really good!
AI would really like this (docs being within code repository).

We are not the first discussing this question, but quick search did not yield previous issues.

But found official docs about readme file:

• General: https://www.drupal.org/docs/develop/managing-a-drupalorg-theme-module-or... →
• Readme: https://www.drupal.org/docs/develop/managing-a-drupalorg-theme-module-or... →

1) There are many AI solutions, many will die off (such as go bankrupt), many other new will be invented, some will grow to take the market share and live on - how AI core will deal with deprecations after it turns out some module no longer useful/usable/inferior for most people, or then better alternatives emerge (alternatives to dreamstudio ✨ Discussion: Add Dreamstudio Provider into core Fixed for example, or ollama 💬 Can a module be provided for use with Jan.ai as a provider? Closed: won't fix ). Is it good idea to maintain this add/remove question in core constantly long time?

2) Drupal Core never had integrations by default, so it is important to know what Drupal CMS leadership thinks about it. The maintenance of services bundled with AI module would become their responsibility (or headache?) as well by extension of being part of Drupal CMS, because of being tightly bundled within AI module.

As alternative, integrations could be bundled out-of-the-box by adding or removing composer dependency. It is same experience from user perspective and it is easier to add/remove. The difference for developer would be more namespaces to develop/track/maintain, while it is only one project today.

I would like this issue to be open for some time.
I don't have good understanding why providers exists within core in the first place.

:-)

Marcus_Johansson → credited mindaugasd → .

practical for people that really just hobby coders doing some changes

Hobby coders in the future will code with AI help, where AI should access AI module documentation without any hassle and produce correct and high quality object oriented code.

This is what I will continue to experiment during dev days :-)

And that is why I am focused on #3455852, because this way AI can have the documentation easily accessible.

I'll try to make a submodule that does this on the dev days

👍

Also @wouters_f, it would be useful to have proposal how you would like ai() to work. For example, how one would choose the provider (openai, mistral) or model? What exact parameters and what response? What use-case would it be best be limited to?

However, how does ai() look with objects today?

Like this? (I "copied" from video)

  $service = \Drupal::service('ai.provider');
  $provider = $service->getInstance('openai');
  $messages = new ChatInput([
    new chatMessage('system', 'You are helpful assistant.');
    new chatMessage('user', $question);
  ]);
  $message = $ai->chat($messages, 'gpt-4o')->getNormalized();
  $entity->set('field_text', $text->getMessage());  

📌 Add Readme Active I think is priority, video was difficult to find and "copy".

Also ->getNormalized() is better to be ->getMessage()
and ->getMessage() is better ->getText()
:-)

Object oriented code can be quite extensive and flexible:

$ai = $provider->textToImage('A cow with earrings');
$entity->set('field_ref_media', $ai->getAsMediaReference());
$entity->set('field_ref_file', $ai->getAsFileReference());

$ai = $provider->chat($messages);
$entity->set('body', $ai->getResponseAsString()); // ?

// etc. etc.

While ai() function could only have few limited uses.
When reading the code, it is not exactly clear what values can it accept or what response can it provide, unless you memorized it.
While object oriented code I demonstrated above can be flexible and self understood.

Yes, it can. LLM provider have been recently incorporated to https://www.drupal.org/project/ai → module.
This is a list of providers it already has: https://git.drupalcode.org/project/ai/-/tree/1.0.x/modules/providers?ref...
To include jan.ai as well, one has to copy one of these modules and modify to jan.ai specifications, then release it as a new module.
Another option is discuss this in this issue: 📌 [meta] Discussion: what LLM providers to include Active

Well done.

Part "Detailed Blog Post" is better to remove and link to the project page instead which contains latest constantly evolving education/marketing materials. While Readme should be likewise also constantly up-to-date about features, with the same commit as new APIs are implemented, so developers could immediately know them. So both types of content separated will be extensive and up-to-date.

Automator part is good, but does not really exist yet, so could be removed and be on project page.

Some more examples:
https://git.drupalcode.org/project/commerce/-/blob/8.x-2.x/README.md?ref...
https://git.drupalcode.org/project/key/-/blob/8.x-1.x/README.md?ref_type...
https://git.drupalcode.org/project/gin/-/blob/8.x-3.x/README.md?ref_type...

If module is very complex like 'commerce', then developer documentation is also moved out completely. And if simpler API module like 'key', then it has it. AI module should have it, because of being so fresh. And I would add a lot of different examples for developers of all APIs they can use.

@abhiyanshu_rawat as Marcus have said, if you haven't used the module, it can be a challenge to fulfill these points...

• In merge request I think its too verbose. All URL pages don't need to be mentioned, maybe the starting point only and user will find the rest. Also try to avoid repeating the same what is already written in module page
• Readme core goal is to provide useful info how to actually use the module. I would like to see there documentation of APIs for developers (code examples).
• Also it does not mention dependencies and requirements, but says user can download the module and use it, but that means, user would run into serious issues. So requirements have to be listed, and to be recommended to install with composer or experiment the installation with project browser.

I think I can give example of my own module here 😁 because its covers these points https://git.drupalcode.org/project/aiprompt/-/blob/1.0.x/README.md?ref_t...
But also look to other examples.

Related parent issue ✨ Integrate with LLM provider Active
When that issue is completed ✅, this issue will be automatically completed as well ✅

AI abstraction layer is complete

I have the same issue on my module 📌 Chat in the sidebar (drupal off-canvas support) Active .
Summary:
* Study how webform module has done it, because it looks very good compared to default.
* Also note that other Drupal features may want to use off-canvas at the same time.
Overall off-canvas is great to do.

@Grienauer also a square logo,
one example what Kevin suggested in #7

I found in multiple sources release date of 10.3 is June 17, so already past the deadline and should be released shortly.

While Drupal 11 will be later (by some sources even maybe December, or maybe July).

I am marking as closed, because release should be soon and because of attributes.

AI may be the module all other will depend on, so a go-to solution to build anything on top.

It can be promoted module already has AI abstraction layer and other features to empower developers to build their apps. Also allows to quickly prompt AI for non developers.

Maybe like I do on my project pages, what already exist, and what is planned, what is consider. Maybe its not the best example, but its clear what to expect. For now AI module is already usable for as a developer framework to base contrib modules on top.

The AI module serves as the cornerstone for developing a wide range of AI-powered applications, from simple automations to complex AI systems. It offers essential tools, immediate out-of-the-box functionality, and supports various AI providers through a flexible abstraction layer. This versatility makes it the go-to solution for any AI integration in Drupal

I asked AI to generate some more good ones, but I am not polishing too much, because those features are not here yet:

• The AI module provides the foundational tools needed to build powerful AI applications, from simple to complex, with minimal to no coding required. It offers plug-and-play functionality and supports multiple AI providers through a unified abstraction layer.
• AI Search enhances your content discovery by understanding the meaning behind search terms through Semantic Search. Utilize an LLM Chatbot to explore, retrieve, or answer queries about your content. [... also add another sentence with keywords for developers like embeddings, vector etc.]
• AI Automators integrate seamlessly with Drupal fields, allowing you to use LLMs to populate and modify any field. Create simple or complex applications by chaining prompts together. Extend functionality with ECA and Widgets to develop comprehensive workflows or AI-powered content widgets.

But I also very interested in this issue, because I am to add a separate section on initiative page as in the draft (above the pyramid) #3372486: Project description → But it also needs a square logo as Kevin described here #3453674-7: Logo for the project page →

mindaugasd → created an issue.

Reworked "Creating and sponsoring new Drupal AI modules" section and renamed it to "Largest projects in development". Added https://www.drupal.org/project/ai → project there. It is a draft.

I moving all description of the project to this issue instead of just solutions. Its good to have a draft.

Not sure what "for the AI Web Connect group" means, so miss info what to comment.

I have questions about the name "AI External Moderation" what does it mean from the user perspective: why it has to be a separate module and be called that way? Could it be included under ai_security module already?

Assuming project_browser enables to install modules with a few clicks (no composer terminal required), then any service could be not included.

There could be specialized UI which hooks into Drupal update mechanism (I am still to learn how that magic works).

Assuming project_browser enables to install modules with a few clicks (no composer required), then any service could be not included.

There could be specialized UI which hooks into Drupal update mechanism (I am still to learn how that magic works).

Related issue 📌 [META] Test Project Browser on hosting providers Drupal people are using Active

Some are very tiny

What can one do with a tiny model. Maybe some specialized automation.

worth whatever cost

Does it need to be included in Drupal CMS for everybody then. For clients who have enough resources for this, agencies/developers can set it up for them.

Gabor was inspired

In general, people show demand to experiment with local AI. So if there is demand for whatever reason, it could be included.

included in Starshot

Another question is how to make it easy enough and accessible to regular Drupal CMS users. How does one install it on the server actually. How much knowledge, investment and experience does it require.

Few more tasks:

• Prepare information what kind of hardware is needed to run ollama, and how much does it cost
• Document how to, or code a feature to shutdown GPU instance when ollama is not in use

Completing these tasks, we could figure how many people can afford this in practice, and how cost effective it can be to run.

One real use-case of this module: installing it locally for people who have a decent GPU on their local computer.

Because of these constraints, it should probably be outside of AI module (not included), unless we find out that it can be practical for most people.

Marcus_Johansson → credited mindaugasd → .

Marcus_Johansson → credited mindaugasd → .

Marcus_Johansson → credited mindaugasd → .

Marcus_Johansson → credited mindaugasd → .

Marcus_Johansson → credited mindaugasd → .

No no, all is fine, thank you for organizing a lot!
All of these events above are also organized by Jamie initiative, including the very first one.

bundles is a very generic name

Yes, and name also clash with Drupal entity bundles.

not extedable and can't really have any functions attached to it

Yes, having it with functions will be better.

In my code code, I abstracted further deep under the name "output" back then, also with functions, examples:

namespace Drupal\custom\Outputs;

class textOutput extends baseOutput {

  public function provides() {
    return ['text'];
  }

  public function setText($text) {
    $this->data['text'] = $text;
  }

  public function text() {
    return $this->data['text'];
  }
}

namespace Drupal\custom\Outputs;

class openaiModerationOutput extends baseOutput {

}

(the same as baseOutput)

namespace Drupal\custom\Outputs;

class openaiChatOutput extends baseOutput {

  public function provides() {
    return ['role', 'text', 'completion_usage', 'completion_model'];
  }

  public function role() {
    return $this->data['response']['choices'][0]['message']['role'] ?? NULL;
  }

  public function text() {
    return $this->data['response']['choices'][0]['message']['content'] ?? '[Result not found]';
  }

  public function completion_usage() {
     return $this->data['response']['usage'] ?? NULL;
  }

  public function completion_model() {
     return $this->data['response']['model'] ?? NULL;
  }

}

namespace Drupal\custom\Outputs;

class stabilityaiImageOutput extends baseOutput {

  public function provides() {
    return ['imageURLs', 'count'];
  }

  public function imageURLs() {
    $urls = [];
    foreach ($this->data['response']['artifacts'] as $key => $val) {
      $urls[] = $val['url'];
    }
    return $urls;
  }

  public function count() {
    return count($this->data['response']['artifacts']);
  }

}

etc.
But 'text', 'role', 'text', 'completion_usage', 'completion_model', 'imageURLs', 'count' as basic data types don't have functions.

For instance an ElevenLabs TextToSpeech would require you to take a choice of the voice you want to use

Azure text-to-speech accepts SSML XML document, where one can define full blueprint everything and in detail of what Azure has to produce: 10 different voices if needed each saying a different thing in a different intonation in different language at specific times, adding pauses, sounds, music etc. It is an old API available for a decade, but highly flexible.

By default it could just accept voice and text, but provider could have a separate and unique helper methods to do various common tasks with a specific API. For example, doChatToSpeech (narrate multiple voices talking with each other), or doSsmlToSpeech (where anything would go).

---
These are my first review for now, I will look into this further.

There are quite a few false positives.

Some examples, where they are creating lists of what warnings to ignore:
https://www.drupal.org/project/commerce/issues/3405463 📌 Enable PHPStan Fixed
https://www.drupal.org/project/simple_address/issues/3436598 📌 Setup GitLab CI pipelines for phpcs, phpstan, eslint, composer validations Fixed

https://git.drupalcode.org/project/commerce/-/blob/8.x-2.x/phpcs.xml?ref...
https://git.drupalcode.org/project/commerce/-/blob/8.x-2.x/phpstan-basel...
https://git.drupalcode.org/project/commerce/-/blob/8.x-2.x/phpstan.neon?...

Maybe its getting better in time.

Added an issue about accessibility

Leveraging Large Language Models (LLMs) to automate accessibility tasks represents a transformative advancement in digital inclusion efforts. These powerful models can replace manual accessibility work by generating image descriptions, providing real-time closed captions, and seamlessly translating content into multiple languages, reducing the burden on content creators, designers, developers, accessibility, and quality assurance teams. As LLM use evolves, it holds the promise of significantly enhancing accessibility and making digital content more inclusive for individuals with diverse assistive technology needs.

Pushed a merge request for using Drupal client, so middleware would work.
It allows me to do anything to request and response (change any setting: timeout, agent, log etc.)

mindaugasd → made their first commit to this issue’s fork.

mindaugasd → created an issue.

Rewritten description changing from "LLM provider" to LLM abstraction layer of "AI" module.

And for output in unreleased code:

  public function setImage(string $image_url, ?string $key = NULL): void {
    $this->content[$key]['image']['url'] = $image_url;
  }

This is the normalization place which happens within AI Chat module, for saving output to the database:

class Message extends MessageBase {

  /**
   * {@inheritdoc}
   */
  public function setResponseData(array $data):void {
    parent::setResponseData($data);

    // fill this message object with data got from API response
    if (empty($this->getId())) {
      $this->setId( $data['id'] );
    }
    $this->setCreated( $data['created'] );
    $this->setText( $data['choices'][0]['message']['content'] );
    $this->setRole( $data['choices'][0]['message']['role'] );
    $this->removeFlag('error');
  }
}

From https://git.drupalcode.org/project/aichat/-/blob/1.0.x/src/Message.php?r...

And for input:

    foreach ($history as $history_message) {

      $content = $history_message->getText();

      if (empty($content)) continue;

      $history_array[] = [
        'role' => $history_message->getRole(),
        'content' => $content
      ];
    }
    return $history_array;

From https://git.drupalcode.org/project/aichat/-/blob/1.0.x/src/Plugin/AIChat...

For not yet released experimental version implementing ✨ Support vision Active there is more complex code:

    foreach ($history as $history_message) {
      $content = $history_message->getContentOpenAI();

      if (empty($content)) continue;

      $history_array[] = [
        'role' => $history_message->getRole(),
        'content' => $content
      ];
    }
    return $history_array;

and

  public function getContentOpenAI() {
    $content = $this->getContent();
    $result = [];
    $types = [];
    foreach ($content as $key => $row) {
      foreach ($row as $type => $val) {
        $types[] = $type;
        switch ($type) {
          case 'text': $result[] = ['type' => 'text', 'text' => $val]; break;
          case 'image': $result[] = ['type' => 'image_url', 'image_url' => $val]; break;
        }
      }
    }

    // get text string in case there is only text
    $types = array_unique($types);
    if (count($types) === 1 && $types[0] === 'text') {
      return $this->getText();
    }

    return $result;
  }

Added information about the to-do items to the issue description

Ideas for next newsletter issue:

• New initiative https://www.drupal.org/project/ai → to merge many modules and prepare AI features for starshot →
• Curated list of interesting issues within https://www.drupal.org/project/ai → module issue queue.

mindaugasd → created an issue.

Very nice Drupal AI security module was proposed: ✨ [META] Create an AI Security module for custom moderation calls Active
A practical work within Drupal in this field.

@solideogloria thanks for the links.

Done

Indeed not a bug :-)

I am moving issue to https://www.drupal.org/project/ai →

Issue description will need an update after further discussion.

We just had discussion about widgets in this issue 📌 Create simple way to iterate for rule Active , which produced some very interesting ideas. Summary:

• expose backend configuration to the frontend (as widgets and popup modals)
• anyone could be able to create any user interface of AI tool they need.
• user interfaces could probably be shared and installed between developers using Drupal recipes.

Sounds like great idea 🙌🏻, security against LLM hacks is important.

This forgets complex prompts which I use all the time. Somehow I need to prove they are very important :-)

As long as it has a way to tweak the prompt and execute the workflow without multiple page loads

Yes, it should expose backend configuration to the frontend, so you could tweak settings as you execute interpolation itself. All within the same form, or in popup modals where needed (assuming I am not missing important details).

That form could even be sorted by weight, for developer to follow and test all the execution workflow.

But maybe this could work architecturally to solve your problem:

developer could create a separate "form display" called "for developer"

layers of the rules

integrating with the rest of the rules

I might not be getting it, because I don't have enough experience with interpolator.

@nicxvan sorry to hear that.

Developer is a user, when using (prompting) AI. I think it it important (and doable) separation architecturally.

Since you mentioned OpenAI playground, I am referencing that you can build such a tool with widgets. So, a similar separate tool would exist in Drupal as in OpenAI, built using Drupal flexible underlying building blocks,

I'm giving text into the prompt as context and trying to get JSON back. When I work in the playground that iterative feedback loop is quick.

Since you are solving a specific problem, could you solve it in the "playground" as you say it works quick, and then come back to Drupal doing the rest of things? By extension, we can replicate this "playground" in Drupal as it is in OpenAI with this architecture, but it would be outside of interpolator configuration side of things?

Another improvement to the idea: developer could create a separate "form display" called "for developer", where developer could expose the widgets and controls needed to test the interpolation on the frontend side, this way having it separated from general workflow developer might be creating for the real user as well.

There is new Drupal AI event coming up
https://www.drupal.org/community/events/drupal-ai-meetup-2024-06-19 →

"Prompt widget" can expose more powerful prompt configuration to the user side, assuming if that would be common use-case, to avoid "so many moving parts".
There should be only few concepts a person has to understand. Like "widgets" could be one of them if we manage to group most common functionality under it.
I think, you should create "AI tool" for yourself for testing, expose widgets you need to the user side, and use it in the simple way.
And this way, anyone could be able to create any tool or interface they need. These user interfaces could probably be shared and installed between developers using Drupal recipes.

Yes, I am doing the same as for other issues and moving it to AI initiative project.

Yes, the same boat as another issue, I am moving it to AI initiative project.

Yes, it served its purpose and no longer fits Drupal core /ideas issue queue, and also does not fit defined process, I am moving it to AI initiative project.

One way you could do it:

• Create a dedicated text field "field_prompt" and have it on the user side, so you could constantly edit it
• And in the backend: configure to interpolate this field with another field as prompt.

Ideally there would be an interface where you could give the input, have a prompt, set which rule / field it should follow, then a button to trigger the call.

I feel not sure about this.

This is developer side:

• Edit field setting prompt
• Save field setting

This is user side:

• Edit content with the field
• Resave content

(User side can be optimized by the developer creating UX for a particular app and use-case)

Maybe you could expand more (for example, your exact example how you are using this), or better, probably Marcus will comment more since having more (most) experience with interpolator.

Thanks for referencing issue about numbers for avoiding going off topic, since I almost did :-)

Changing names means confusion.
While creating new product means it is easier to experiment with new features and be more relaxed about it. Introducing no change to existing workflows and understanding.
And leaving all the other names as it is also means less change and less confusion.
Drupal 7 is deprecated anyway. Everyone knows it as Drupal 7, I talk about it using this name all the time, so it is good to leave it just as it is.
Dropping numbers from Drupal is also a big change (such as rolling release). It is easier to communicate to the client, that version has changed (like Drupal 11), and now the Drupal major version need to be upgraded until a specific date.

List of Drupal AI modules from 1st comment I moved to separate issue so that updates to the list could be tracked: #3429343: Overview of Drupal AI modules in one place →

@alexmoreno thanks.

Adding new modules to the list, but also moving https://www.drupal.org/project/ai → module from 'empty' category to the top of the page.
We have #3447196: Taking over the AI namespace → to prepare official Drupal AI solution collaborating across many teams and merging many solutions, as well as basing them on LLM abstraction layer 📌 Create LLM abstraction layer Active .

I created a new issue to possibly move this module overview page to new ✨ Create AI ecosystem "add-ons" page Active of AI module.

Added new modules to the list:
https://www.drupal.org/project/unstructured →
https://www.drupal.org/project/tavily →
https://www.drupal.org/project/auphonic →
https://www.drupal.org/project/fireworksai →
https://github.com/ivanboring/ai_interpolator_youtube
https://www.drupal.org/project/ai_interpolator_google_places →
https://www.drupal.org/project/extensionbot →
https://www.drupal.org/project/ai_translator →
https://www.drupal.org/project/augmentor_azure_openai →
https://www.drupal.org/project/mistral →

Also changed interpolator description and grouped all its modules.

vote for colorful one:)

Even more:

Drupal Express
Drupal QuickStart
Drupal Ready
Drupal Suite
Drupal Stellar (starshot:)
Drupal Nova
Drupal Boost
Drupal Elevate

Nice https://www.drupal.org/project/mistral →

"Website" sounds nice.