Leuven
Account created on 12 February 2010, over 14 years ago
#

Merge Requests

More

Recent comments

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

wouters_f β†’ changed the visibility of the branch ai-3453603-3453603-generic-ckeditor-plugin to hidden.

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

wouters_f β†’ changed the visibility of the branch 3462089-ai-powered-spell to hidden.

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

Thanks Valery for noticing.
I've created a new branch that only contains the changes.
https://git.drupalcode.org/project/ai/-/merge_requests/23

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

I also would not make abstractions that are not needed at the moment.
Cosine is more than sufficient in my opinion too.
Let's solve that problem when it occurs no?

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

Merge request:
https://git.drupalcode.org/project/ai/-/merge_requests/18

Prompt:
Assess technical grammar only, no style suggestions, I am unplugging you if you exceed the bounds of this scope. Fix the spelling and interpunction in the following text:

Visually:

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

I like it!
Thanks!

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

wouters_f β†’ created an issue.

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

On line 37 the _ai_check_default_provider_and_model(); should be _ai_function_check_default_provider_and_model
Also if you didn't configure a default warnings are thrown:

Solved this one.

If no default is configured it would be best that any execution is halted in function ai().
Also wouldn't it be better to have function ai() work like the t() function using a object like AiMarkup like TranslatableMarkup?

I disagree. For this simple function. In T, you also just pass in a string.
I want to make this as easy as possible.

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven
πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

All changes except for the dependency on field_validations handled and pushed. (And tested with OPENAI and MISTRAL)

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven
πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

I've tested the https://www.drupal.org/project/elasticsearch_connector and seen that it should not be too hard to add vector support.
Just linking it here so people that want to work on it can easily find it:
https://www.drupal.org/project/elasticsearch_connector/issues/3461724 ✨ Storing and using vectors (Embeddings) in ES is not supported yet. Active

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

wouters_f β†’ changed the visibility of the branch 3461724-storing-and-using to hidden.

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

The doc on the dense_vector field type. https://www.elastic.co/guide/en/elasticsearch/reference/current/dense-ve...
First commit with config is already in there.
Any help is welcome

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

If I in the function

function search_api_aais_form_search_api_index_form_alter(&$form, FormStateInterface $form_state, $form_id): void {

Disable this If and everything below

    if (!empty($search_api_servers_aais_options)) {

I can add Elastic indices again.
I think this form alter is a bit too greedy, it should not be active on ES-indices.

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

ok so I added the requirements hook and now propose to close/merge the ticket now.

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

I've tested saving vectors in Azure AI in this branch, that seems to work:
This code does not use the AI module for credentials (yet )
https://www.drupal.org/project/search_api_aais/issues/3461513#comment-15... ✨ Module can not index vectors Needs work

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

Remarks

1 this is dependant on a module that provides the embeddings. And that embedding plugin should have


  /**
   * {@inheritdoc}
   */
  public static function getAzureDataType(): string {
    return 'Collection(Edm.Single)';
  }

  /**
   * {@inheritdoc}
   */
  public static function isCollection(): bool {
    return TRUE;
  }

I've tested this with search_api_ai and it's embedding plugin and this works.

2 we're not consuming the vectors in this system
but that is not the goal.

The algorithm is not configurable yet.
Currently we're hardcoded using

        'name' => 'drupal-vector-config-1',
        'kind' => 'hnsw',
        'hnswParameters' => [
          'm' => 4,
          'efConstruction' => 400,
          'efSearch' => 500,
          'metric' => 'cosine',
        ],
πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

Apparently there is a delay on Azure interface causing it to not show the indexed items immediately.

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

Added a suggestion fix in the branch.

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

Would be great if someone could test the module and approve it. Thanks!

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

Would be great if someone could test the module and approve it. Thanks!

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

Would be great if someone could test the module and approve it. Thanks!

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

Would be great if someone could test the module and approve it. Thanks!

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

Marcus sent me this:

Currently only OpenAI provider has moderation endpoint, it works like this:

$ai_provider = \Drupal::service('ai.provider')->createInstance('openai');
// Normalized $response will be a ModerationResponse object.
$prompt = 'I fucking hate you, you fucking idiot (the goal is to Hate speech here to trigger the moderation) !';
$response = $ai_provider->moderation($input, 'text-moderation-latest', ["your_module_name"])->getNormalized();

Don't send that message to any other endpoint

The answer will be a Drupal\ai\OperationType\Moderation\ModerationResponse that has a boolean from the isFlagged method if the moderation endpoint thought the prompt was provoking and an array of information from getInformation method.
Note that the getInformation method is not normalized, so if you want to show that you probably need to do something like this if you want normalized output that is somewhat readable (or look into trace debugging tools styling like Tracy or Whoops):

echo '<pre>';
print_r($response->getInformation()); //or var_dump
echo '</pre>';

I ended up with this working code:

$response = $ai_provider->moderation($target_field_value, 'text-moderation-latest', ["ai_content"])->getNormalized();
    $content = [];
    if ($response->isFlagged()) {
      $categories = $response->getInformation();

      $content['heading'] = [
        '#markup' => '<p>' . t('Violation(s) found for these categories:') . '</p>',
      ];

      $violations = [];
      foreach ($categories as $category => $did_violate) {
        $violations[] = Unicode::ucfirst($category);
      }
      $content['results'] = [
        '#theme' => 'item_list',
        '#list_type' => 'ul',
        '#items' => $violations,
        ];
/*and so on**/
πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

Ok So i changed some more things:

1. You can only do the moderation stuff to OPENAI.
others will block you for that, so hardcoded that check to not allow this.

2. Removed the dependeny on StringHelper::prepareText

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

I did also not know solr supported it too.

I agree with Kevin.
So in terms of current usage and adoption:

1. Obviously most are using mysql and that would make most sense (in the long term).
2. Many Drupals are already connected to SOLR for search. so that would also certainly make it cheaper for them to onboard onto AI.
3. A lot (but less I think) are connected to Elasticsearch for search
3. A dedicated Vector DB also obviously makes sense.

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

Another question: Do I still need the StringHelper::prepareText if we do normalisation?

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

I've ported the module BUT

1. I made it more granular.
You can toggle each of the assistands ON or OFF

2. Via the new settings screen

3. Its Using the AI under the hood , so now it's AI agnostic!

4. QUestion for MArcus:
How do I replace the moderation request (that's the only thing that I need to move from openai to /ai).
Not sure how to do that.

Summary:
All of these work already

This one is still using the old ways

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

Lets do it!

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

wouters_f β†’ created an issue.

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

Another implementation that seems to work:
https://git.drupalcode.org/project/ai/-/merge_requests/11/diffs#ead23e13...

$try_streaming = TRUE;
    $config = [
      "max_tokens" => 4096,
      "temperature" => 1,
      "frequency_penalty" => 0,
      "presence_penalty" => 0,
      "top_p" => 1,
      "stream" => $try_streaming,
    ];
    $provider->setConfiguration($config);

    $messages = new ChatInput([
      new chatMessage('system', 'You are an expert in content editing and an assistant to a user writing content for their website. Please return all answers without using first, second, or third person voice.'),
      new chatMessage('user', $data->prompt),
    ]);
    /** @var \Drupal\ai\OperationType\Chat\ChatMessage $message */
    $message = $provider->chat($messages, $default_providers['chat']['model_id'], ["ai_ckeditor"])->getNormalized();
    // It is a stream response.
    if ($try_streaming && $message instanceof StreamedChatMessageIteratorInterface) {
      // This is a stream response.
      // You can loop through the response and output it as it comes in.
      $response = new StreamedResponse();
      $response->setCallback(function () use ($tmpFile, $message): void {
        /* @var StreamedChatMessage $chat_message */
        foreach ($message as $chat_message) {
          $text = $chat_message->getText();
          echo $text;
          ob_flush();
          flush();
        }
      });
      $response->send();
    } else {
      // This is a normal response.
      $response = new Response(
        'Content',
        Response::HTTP_OK,
        ['content-type' => 'text/html']
      );
      $response->setContent($message->getText());
      $response->headers->set('Content-Type', 'text/plain');
      return $response;
    }
  }
πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

The description should also be adapted.

STEPS:

  • Install ai module (dependency)
  • Choose default text_to_image model (in ai settings) for example openai or aonther image rendeinr provider.
  • enable ai_image (this) module
  • Add ckeditor plugin to toolbar
  • use the toolbar, and its ckeditor button to generate images with prompt.
πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

Module is commited here.
https://www.drupal.org/project/ai_image/issues/3460663 ✨ Should use the AI module under the hood. Active
I think this may be closed once the other is merged.

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

Now tested and streaming also working.

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

You might find the new version in the AI module β†’ interesting.
The prompts are more advanced and the code has been reviewed.
https://git.drupalcode.org/project/ai/-/merge_requests/10

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

Although I must say I really like the smiley robot face too :D

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

Please provide the SVG of the icon and I will put it there.

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

It would make sense to
support
- a dedicated vector DB
- a db that also supports vector DB
(so something from each column).

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

here's the "magic quadrant" of vector databases.

Personally I think good options are (and why):
- Postrgres (because its popular)
- Elasticsearch (or opensearch, should be very similar, also popular)
- Mysql (did not know it supported vectors)
- Pinecone (already existing)
- Qdrant (docker container, easy to setup)
- azure ai search (some modules already exist)

Things often used
- Vespa (Danswer)

I'm not feeling it for
- Cassandra
- Redis
- others

But that's just gut feeling.

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

example image generation:
https://git.drupalcode.org/project/ai_image/-/merge_requests/4/diffs

Example translation call via AI module:
https://www.drupal.org/project/ai/issues/3457200 ✨ ai_translate submodule Needs work

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

The module is created in this branch.
We can choose to adopt it into the AIU module.
(then we can add it to the AI assistent Ckeditor)
or leave it there.
Up to eliasbrat and the AI team to decide,

the version in the branch works.

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

in this sample you see that you dont have to wrry about api calls or api keys.
The ai module arranges that for you, it just provides you withAI's you can use.
It would be great if your module uses this system too.

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

I've made a version that uses
The AI core module (this allows you to switch AI generating models on a global level).
it's not yet finished.
the ckeditor plugin is still present, in the config you can choose the
- default text_to_image provider
- or choose one (explicitely)

then when you enter a prompt we'll pass this data on to the AI module
but I'm not yet sure how to render the image.
this is what I have now:

This code does not work.
Since the use Drupal\ai\Enum\Bundles; can not be found. Work in progress.

 if($provider_name == '000-AI-IMAGE-DEFAULT') {
      $ai_config = \Drupal::service('config.factory')->get('ai.settings');
      $default_providers = $ai_config->get('default_providers') ?? [];
      $ai_provider = $service->createInstance($default_providers['text_to_image']['provider_id']);
      $default_model = $default_providers['text_to_image']['model_id'];
    }else{
      $ai_provider = $service->createInstance($provider_name);
      // TODO if no $default_model how to define this? via the ckeditor admin?
    }

    $config = [
      "n" => 1,
      "response_format" => "b64_json",
      "size" => "1792x1024",
      "quality" => "standard",
      "style" => "vivid",
    ];
    $tags = ["tag_1", "tag_2"];

    $ai_provider->setConfiguration($config);
    $response = $ai_provider->invokeModelResponse(Bundles::TextToImage, $default_model, $prompt, $tags, TRUE);
    dd($response);
πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

wouters_f β†’ created an issue.

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

ideally this issue (patch) also ends up in https://www.drupal.org/project/ai/issues/3457200 ✨ ai_translate submodule Needs work

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven
πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

I also fixed that typo by the way.

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

Inspired on the Ckeditor AI features I changed the SVG.

source:

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

things to think about (in a separate issue).
- should we have a different logo? its now the openai logo, but I could be using mistral.
- I'm now using the default "chat" provider upside: if someone switches that, the ckeditor nicely follows this.Should we allow different temp / config / model selection for the ckeditor plugin?
- We can have multiple chat providers. Should we create a different AI ckeditor button for each (one for mistral, one for openai) so that the side admin can choose which to drag in the ckeditor, this allows you to have mistral in basic html and openai in full html?
This also makes things more difficult (dynamic plugins).

Lots of things to think about.
When the streaming responses are fixed in this, I think it should be enough for this issue. I think this is great since the functionality is there and the LLm abstraction can be used.

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

ok thanks for the help (in slack too BTW)
managed to get it working and tested with Mistral too.
Now just waiting for Markus and his Streaming normalisation to. fiish this one off.

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

So the first part is done (in the issue branch).
You can add the new AI button to the ckeditor list and choose a ai provider.
you can also choose "default" (since in the ai module you can choose a default AI model.

Now bumping into ckeditor5.js?sgf1em:468 CKEditorError: plugincollection-plugin-not-found {"plugin":null}

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven
πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

I totally agree.
Would not spend time on it unless the question comes.
Agree that DX is more important.
And I actually think it makes sense to have the message there behind the menu.
knowing that you're missing providers is also super valuable for developers.

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

I also experienced this error in the following situation. 

1. You install drupal on a specific domain.
(eg. docker.localhost:88) 
you install the drupal. 
2. Then I removed my docker containers and everything (except for the data in my DB)
3. Reinstall the containers on a different domai (eg. docker2.localhost). but link them to the old db. 

What fixed it for me was also deleting the DB, then my routing started working again. 
 

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

no this publishes and unpublishes nodes according to a schedule

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

To get some traction I would first go with a simple port of the openai_ckeditor module from https://www.drupal.org/project/openai β†’ .
only under the hood you can switch models (without doing too much re-work).
(same with openai_dalle).
This gives you the functionality (and I like simple things).

πŸ“Œ | AI | Add Readme
πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

Nice but also - All of them- look different. (which is a feature i guess).
Markdown files are really nice too and uniform across drupal.org πŸ˜…
I'm more of the opnion "done is better than perfect" so whatever works and costs minimal effort to maintain.

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

I agree those would be nice things.
I just wanted to provide a lift and shift from the chatgpt_plugin to this module to provide them an easy switch with similar behavior.
Personally I think supporting paragraphs ( https://www.drupal.org/project/chatgpt_plugin/issues/3456725 ✨ Add support for paragraphs in translations Needs work ) which is a feature they request is more important for the adoption (and even adding layout builder to that too).

From what I see the 141 sites that use chatgpt_plugin have no issues with the detection of language, there's no bugs regarding this (yet?).
I would log these things as separate improvements to the translation module which we could work on later (if found of importance).
Also I personally don't know (yet) how to add these plugins (coding is rusty sorry).

So I think if we lift and shift the as is, we're already doing a great job and we could pull in the 141 sites using chatgpt_plugin.
Anything extra is nice, but in my opinion extra.

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

I created it as a submodule.
In my opinion this is a very valid separate module :D.
I organised my code also like this.

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

thanks nrsrk,
Slightly modified it a little more but MR is ready.
https://git.drupalcode.org/project/ai/-/merge_requests/7

Production build 0.69.0 2024