Contrib.social

Providing images for prompts

Open on Drupal.org →
Created on 7 November 2024, about 1 month ago

Problem/Motivation

  • Images provided via the AI chat explorer in the AI module are not recognized from multiple models checked through the Ollama Provider.
  • The model responds with a message about no image being detected. For example: I'd be happy to provide a caption for you! However, I don't have any specific image or context in mind. Could you please provide more information about the image you're referring to (e.g. what's happening, who/what is in it)? Or do you want me to generate a generic caption?

Please see the attached screenshot for an example.

Also note that I was getting similar results when using https://www.drupal.org/project/ai_image_alt_text → . In that case, the model responded with "I'm ready to provide alt text. What is the image?"

Please see the second screenshot for an example of that.

Steps to reproduce

  1. After configuring Ollama, make sure the AI API Explorer module is also enabled and visit /admin/config/ai/explorers/chat-generation
  2. Use a visual AI model, such as llama3.2-vision:latest
  3. Make sure to include an image
  4. The response will indicate that no image was included with the prompt.

This may be a continuation of https://www.drupal.org/project/ai/issues/3476097 ✨ Add image Chat input possibility with Ollama Active (from before this provider module was separated out from the main AI module. I did confirm that the code added in the MR (https://git.drupalcode.org/project/ai/-/commit/28e95db7d4f767c3628a54b3e...) is present in ai_provider_ollama:1.0.0-beta1.

🐛 Bug report
Status

Active

Version

1.0

Component

User interface

Created by

🇺🇸United States rklawson

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

contrib.social Blog FAQ Discussions
Production build 0.71.5 2024