Module can not index vectors

Created on 15 July 2024, 4 months ago
Updated 16 July 2024, 4 months ago

Problem/Motivation

The module can not index vectors (Embeddings) created by another AI system.

Steps to reproduce

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes

Feature request
Status

Needs work

Version

1.0

Component

Code

Created by

🇧🇪Belgium wouters_f Leuven

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @wouters_f
  • Merge request !1[3461513] allow addition of embeddings → (Open) created by wouters_f
  • Pipeline finished with Success
    4 months ago
    Total: 134s
    #224828
  • Status changed to Needs work 4 months ago
  • 🇧🇪Belgium wouters_f Leuven

    Remarks

    1 this is dependant on a module that provides the embeddings. And that embedding plugin should have

    
      /**
       * {@inheritdoc}
       */
      public static function getAzureDataType(): string {
        return 'Collection(Edm.Single)';
      }
    
      /**
       * {@inheritdoc}
       */
      public static function isCollection(): bool {
        return TRUE;
      }
    
    

    I've tested this with search_api_ai and it's embedding plugin and this works.

    2 we're not consuming the vectors in this system
    but that is not the goal.

    The algorithm is not configurable yet.
    Currently we're hardcoded using

            'name' => 'drupal-vector-config-1',
            'kind' => 'hnsw',
            'hnswParameters' => [
              'm' => 4,
              'efConstruction' => 400,
              'efSearch' => 500,
              'metric' => 'cosine',
            ],
    
  • 🇧🇪Belgium interx Flanders

    Thanks for the report and the MR!

    We have vector fields on the roadmap in a later satge
    The MR is currently not complete/ready for merge, but we'll see how we can integrate it in a generic yet flexible fashion.

Production build 0.71.5 2024