🇩🇪Germany @Marcus_Johansson

Those things are fixed, so I think it mostly works now.

The idea with default models is not bad. Its easy to add to the install config. It's just hard to decide what people wants there. Taking models that usually work for without failing on free tier would be preferred I think.

The autocomplete seems to have a bug when you fill in the actual full text. Will set it back to Needs Work. Also add some bolded information about the access token.

I think the general feeling is not in core.

Its basically there to make sure that anyone creating a third party plugin provider, uses it. This forces them to read about it and understand the issues with just having them in a config.

I think we will have to add basic auth or some other reverse proxy solution to Ollama soonish, because you might want to have setups where Ollama is located on the public internet away from your website. Then the Key module makes sense.

Could we set this to fixed since it worked out well? :)

Marcus_Johansson → created an issue.

Marcus_Johansson → created an issue. See original summary → .

Marcus_Johansson → created an issue.

Marcus_Johansson → created an issue. See original summary → .

Marcus_Johansson → created an issue. See original summary → .

I think each operation type and each provider needs to be documentation, I'm wondering if there is someway to make subpages in markdown from the Drupal documentation, so we don't have to rewrite the same documentation twice.

Merged and thanks for you work!

ollama serve should be the main command to make it start listening. You can run ollama without it, via ollama run, so its a little bit strange.

I added this follow up ticket as good-to-have: https://www.youtube.com/watch?v=LFFoGfYFMn4&ab_channel=DrupalAIVideos

Marcus_Johansson → created an issue.

Should we close this ticket and anything that should go into core needs to be a follow up request?

Yes, its on purpose, check https://www.drupal.org/project/ai/issues/3455954 ✨ Reduce drupal core requirement from 10.3 to 10.2 Closed: works as designed .

The structure is there and I have written the logic for it in OpenAI as well.

Also ->getMessage() is better ->getText()
and ->getNormalized() is better to be ->getMessage()
:-)

@mindaugasd - I agree it should be getText(), its gets even more obvious with this ticket: https://www.drupal.org/project/ai/issues/3456629 🌱 Add Image into ChatMessage object (refactor or no breaking change) Active

For getNormalized(), I think its good to have a known way of receiveing the data that is known to be normalized over all the operations. So you use getNormalized() for the output from text-to-image etc as well, they just happen to be different objects returned with a common interface.

Marcus_Johansson → created an issue.

Added an update hook since the version config needs to be force updated for Sonnet 3.5 to work.

Marcus_Johansson → created an issue.

Its already in DEV, so set to fixed.

I think you will have to eiher input a normalized object as input or the providers input, doing something in between will be really confusing since you now have two normalizing layers, and one that can't do everything that in the spec. In you example with Chat, attaching images or having chat between two humans and one AI or having chat history is not possible.

I would also do the assumption that if someone uses a function like this, they do not want to meddle around with objects, thats what gets them to do procedural code in the first place, so taking raw data and receiving raw data, with the option of normalizing is the way to go I think. People still trying to use Drupal 7 syntax are most likely used to associative arrays, rather then objects.

so you can do something like

$messages = [
  ['role' => 'system', 'content' => 'You are an image reader'],
  ['role' => 'user', 
    'content' => [ 
       'type' => 'text,
       'text' => 'Could you describe the image for me?'
    ],
    [
      'type' => 'image_url',
      'image_url' = base64_encode_helper($someimage),
    ],
  ],
];

ai($messages);

// Returns something like in assoc array
{
  "id": "chatcmpl-123",
  "object": "chat.completion",
  "created": 1677652288,
  "model": "gpt-3.5-turbo-0125",
  "system_fingerprint": "fp_44709d6fcb",
  "choices": [{
    "index": 0,
    "message": {
      "role": "assistant",
      "content": "\n\nThat is an image of a dog",
    },
    "logprobs": null,
    "finish_reason": "stop"
  }],
  "usage": {
    "prompt_tokens": 9,
    "completion_tokens": 12,
    "total_tokens": 21
  }
}

This would of course not be possible to abstract into other providers since its OpenAI specific, but I'm guessing people using this are only looking for a quick fix to call the api.

The other way would be to have ai and ai_normalized, where ai_normalize takes and returns normalization objects, but since its one function for all different calls, it will not be able to suggest whats in that object in your IDE.

Or you could take the decision that its only for chat, but even then things like images (or other files) or conversations for chatbots needs to be solved.

Or its a pure text producing endpoints that takes text in and spits text out?

If you get the latest review hopefully it works better.

Marcus_Johansson → created an issue.

Hi Ressa, I get notifications. Thanks! I'm just out with a cold, but will get back to you as soon as I'm better.

I'll check with the other guys, but I'm sure we can make you a maintainer for the documentation pages.

The steps should be identical - I used Gemma so its a showcase that most modern laptops can run.

@ressa - check out here: https://youtu.be/LFFoGfYFMn4

Marcus_Johansson → created an issue.

I might do a video about it, since with Gemma, it would actually be a showcase for something that you can host on a webserver/your own none-GPU laptop with 16GB of RAM.

Ah, so something like this.

1. Start Ollama with an environment parameter, something like - OLLAMA_HOST="0.0.0.0" ollama serve
2. When you define the hostname in the page /admin/config/ai/providers/ollama it should be http://host.docker.internal. This is a special hostname that connects to the docker parent host.

That should work on Linux and 99% work on Mac. On Windows it would be two commands and some special Windows sauce to setup Ollama, so something like

set OLLAMA_HOST=0.0.0.0
ollama serve
approve the firewall

If it doesn't work on Windows let me know and I'll test on my gaming computer.

Feel free to test.

It was the discovery cache not being cleared, it has been fixed now.

Marcus_Johansson → created an issue.

Its createInstance, not getinstance...

Also make sure to start Ollama so it listens to world in that case, since it only listens to localhost by default and your DDEV machine is not localhost.

https://github.com/ollama/ollama/issues/703

Yes, either you have to setup Ollama in a DDEV docker container or you can try this for hostname:

host.docker.internal

It should work in most cases for connecting to the Docker host.

A use case for our module bot :D

This module has a plugin system for this: https://www.drupal.org/project/field_validation →

We could do a AI field validation module in the AI module, but since field validation is a contributed module, I would opt for this being its own module and using the Field Validation and AI module together.

No need to reinvent the wheel.

Marcus_Johansson → created an issue.

All the discussed models are now in review!

Done, can be tested with chat and embed. More operation types coming.

The models that you setup on the config page are to show when using form selectors, when using calls you can still give any model.

I think its better we just take the company name, otherwise someone else will take the namespace and make some solve-one-issue module and people gets confused when it shows up first on Google as to why that module can't be used with AI module.

Can be tested.

Feel free to test Groq.

First version, feel free to review!

I/we have looked into this already for form validation via YOLO. I think it would make sense to have validators for certain fields. I think it is not for automators, but rather the editorial experience.

In a text field you can have a form widget add-on that adds a validator that scans the text against a prompt - "Is this text leaking any information about our future projects?" and it stops and gives you a validation message if it does.

Or for an image validation for a Mercedes site - "Is this image a Mercedes car, otherwise do not validate".

So its outside of the scope of the automator.

We should look if there is a module with a plugin system that does this today without AI already. Then we can build in AI for it. That would take less then a day.

Building a stable framework for text and image fields it is 2-4 days.

Regarding document how to, or code a feature to shutdown GPU instance when ollama is not in use.

This should be done in a host solution like a Runpod module or something similar. I have been thinking about building such a solution. Anyway, it should not be in the AI module, the external modules can talk to the AI module for events.

So an initial version of this provider is done and can be test on DEV.

The first version has no controlling of Ollama, like pulling/deleting models. This has to be done via command line at the moment. But as soon as that is done, its usable.

Someone should test it with the explorers.

It supports chat and embed for now. Text completion when I get the time to add that.

Doh, https://www.drupal.org/project/ai/issues/3453595 📌 Groq LLM Provider Active

Marcus_Johansson → created an issue. See original summary → .

I don't think we can put them under one submodule. If you want some specific function that happens to fall under one provider, that should be possible.

Fireworks AI has for instance an ImageToImage model that creates creative QR codes from an image. If my module does QR codes and I want that and only that functionality, I should be able to require just that provider.

We have Fireworks support! https://www.drupal.org/project/fireworksai →

Marcus_Johansson → created an issue. See original summary → .

The most important reason is that a production ready version is probably around Drupal 11 release, so 10.3 will have been around for some time.

But its also so we don't have to support annotations at all. Its already built around attributes which makes it easier to understand.

Thanks @szloredan, approved, tested and merged.

Thanks @szloredan - approved and merged into dev.

Marcus_Johansson → created an issue.

Marcus_Johansson → created an issue.

Also since they are just interfaces, not implementations it would be hard to use the plugin system. I added attributes on them however.

And allowing external modules to add them is easy as changing two lines of code. But it would be unfortunate if for instance and ElevenLabs method adds a TextToSpeech, then Deepgram adds its own TextToSpeech etc.

Marcus_Johansson → created an issue.

Testable on DEV.

@abhiyanshu_rawat - this is a completely new module, do you know how it works? How would you write the README?

I'll split this up in new tickets and since some of the stuff above is done without a ticket, this ticket is now an epic/discussion to the tickets below.

Marcus_Johansson → created an issue. See original summary → .

Marcus_Johansson → created an issue.

Marcus_Johansson → created an issue.

Marcus_Johansson → created an issue.

Marcus_Johansson → created an issue.

Also we should add on all of the explorers that doesn't have a provider, even if a provider is installed, a link to the documentation with a list of the providers of a specific type.

Outside of that, I think a warning message in Status report if providers are missing as well.

• Default provider ticket crosslinked here as a separate ticket: https://www.drupal.org/project/ai/issues/3454928 📌 Make default operation providers possible Active
• Documentation with a provider section is needed as well.
• Also a separate ticket for creating a README should be done.