Contrib.social

Automators: \Drupal\ai_automators\PluginBaseClasses\VideoToText::createVideoRasterImages should use \Drupal\ai\OperationType\GenericType\ImageFile

Open on Drupal.org →
Created on 11 March 2025, about 1 month ago

Problem/Motivation

The VideoToText plugin creates an array of images that are base64 encoded strings in \Drupal\ai_automators\PluginBaseClasses\VideoToText::createVideoRasterImages:

    $images = glob($this->tmpDir . 'raster-*.jpeg');
    foreach ($images as $uri) {
      $this->images[] = 'data:image/jpeg;base64,' . base64_encode(file_get_contents($uri));
    }

Whereas \Drupal\ai\OperationType\Chat\ChatMessage expects its $images to be ImageFiles.

  /**
   * The images files in an array.
   *
   * @var \Drupal\ai\OperationType\GenericType\ImageFile[]
   */
  private array $images;

This results in an error when the message is processed - e.g. in \Drupal\ai_provider_openai\Plugin\AiProvider\OpenAiProvider::chat you get:

Error: Call to a member function getAsBase64EncodedString() on string in Drupal\ai_provider_openai\Plugin\AiProvider\OpenAiProvider->chat() (line 318 of modules/contrib/ai_provider_openai/src/Plugin/AiProvider/OpenAiProvider.php).

The Anthropic provider does something very similar:

        if (count($message->getImages())) {
          foreach ($message->getImages() as $image) {
            $content[] = [
              'type' => 'image',
              'source' => [
                'type' => 'base64',
                'media_type' => $image->getMimeType(),
                'data' => $image->getAsBase64EncodedString(''),
              ],
            ];
          }
        }

Steps to reproduce

Set up a workflow like https://workflows-of-ai.com/workflow/automatic-video-editor

I hit the following error:

Error: Call to a member function getAsBase64EncodedString() on string in Drupal\ai_provider_openai\Plugin\AiProvider\OpenAiProvider->chat() (line 318 of modules/contrib/ai_provider_openai/src/Plugin/AiProvider/OpenAiProvider.php).

ReflectionMethod->invokeArgs() (Line: 186)
Drupal\ai\Plugin\ProviderProxy->wrapperCall() (Line: 103)
Drupal\ai\Plugin\ProviderProxy->__call() (Line: 131)
Drupal\ai_automators\Plugin\AiAutomatorType\LlmVideoToVideo->generate() (Line: 70)
Drupal\ai_automators\AiAutomatorRuleRunner->generateResponse() (Line: 70)
Drupal\ai_automators\Plugin\AiAutomatorProcess\DirectSaveProcessing->modify() (Line: 229)
Drupal\ai_automators\AiAutomatorEntityModifier->markFieldForProcessing() (Line: 129)
Drupal\ai_automators\AiAutomatorEntityModifier->saveEntity() (Line: 31)
ai_automators_entity_presave()

Proposed resolution

The VideoToText plugin should add the images as ImageFile objects - perhaps in a similar way to \Drupal\ai\OperationType\Chat\ChatMessage::setImageFromFile

Remaining tasks

User interface changes

API changes

Data model changes

🐛 Bug report
Status

Active

Version

1.0

Component

AI Automators

Created by

🇬🇧United Kingdom mcdruid 🇬🇧🇪🇺

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

contrib.social Blog FAQ Discussions
Production build 0.71.5 2024