Providing images for prompts

Created on 7 November 2024, 7 days ago

Images provided via the AI chat explorer in the AI module are not recognized from multiple models checked through the Ollama Provider.
The model responds with a message about no image being detected. For example: I'd be happy to provide a caption for you! However, I don't have any specific image or context in mind. Could you please provide more information about the image you're referring to (e.g. what's happening, who/what is in it)? Or do you want me to generate a generic caption?

Please see the attached screenshot for an example.

Also note that I was getting similar results when using https://www.drupal.org/project/ai_image_alt_text → . In that case, the model responded with "I'm ready to provide alt text. What is the image?"

Please see the second screenshot for an example of that.

After configuring Ollama, make sure the AI API Explorer module is also enabled and visit /admin/config/ai/explorers/chat-generation
Use a visual AI model, such as llama3.2-vision:latest
Make sure to include an image
The response will indicate that no image was included with the prompt.

This may be a continuation of https://www.drupal.org/project/ai/issues/3476097 ✨ Add image Chat input possibility with Ollama Active (from before this provider module was separated out from the main AI module. I did confirm that the code added in the MR (https://git.drupalcode.org/project/ai/-/commit/28e95db7d4f767c3628a54b3e...) is present in ai_provider_ollama:1.0.0-beta1.

🐛 Bug report

Status

Active

Version

1.0

Component

User interface

Created by

🇺🇸United States rklawson

Live updates comments and jobs are added and updated live.

Merge Requests

!1Providing images for prompts
Open
🇺🇸United States sethhill
updated 3 minutes ago

Issue created by @rklawson
First commit to issue fork.
Merge request !1Modify message structure to match OpenAI format. → (Open) created by sethhill
Pipeline finished with Success
about 6 hours ago
Total: 146s
#339299
Comment about 6 hours ago →
🇺🇸United States sethhill
Since the Ollama provider is using the OpenAI-compatible format, I tried replacing the existing ChatMessage structure with that found in the OpenAI provider. This allowed image alt text generation to work using Ollama as the provider.
Comment 3 minutes ago →
🇩🇪Germany marcus_johansson
Thank you Seth, code looks good. I'm not sure what changed since the last ticket or if it just happened to work with Llava, anyway this is tested both with Llava and Llama and looks good getting merged.

Production build 0.71.5 2024