Contrib.social

Make vector databases abstracted and installable for recipes

Open on Drupal.org →
Created on 27 February 2025, about 1 month ago

Problem/Motivation

Right now we have the possibility for you to create a recipe that applies any 3rd party module for the AI providers with a default operation type model. This means that you can create recipes without necessarily having to bind them to OpenAI or Ollama or anything else.

We should be able to do the same for:

  1. Setting up a VDB providers keys (that might make sense to move to Key module and be generic).
  2. Setting up a server and index, independent on the VDB provider and independent on the Embeddings model.

This means that you should be able to add a default VDB Provider.

This means that you should be able to define a recipe for installing a key like:

    vdb_provider_milvus.settings:
      setupVDBProvider:
        key_value: ${milvus_api_key}
        key_name: milvus_api_key
        key_label: 'Milvus API Key'
        provider: milvus
      simpleConfigUpdate:
        host: ${milvus_host}
        port: ${milvus_port}

And then in your recipe you would be able to prompt in console or Project browser for hostname, port and key.

If no default VDB provider exists, Milvus or whatever gets installed first can opt in to become this, but it should be settable where you change the other operation types.

When that is setup you should then be able to import a server and an index from config, just that you can set default on the database and embedding_engine in the server config.

This should also be possible to do in the form, and when the form is getting saved, if the dimensions is set to set_dimesions = false and the embedding_engine is set to default, we manually collect the dimensions. This has to happen on saving, not in the form, so it will also work on recipe.

The index configuration doesn't really have anything connecting to dynamic values, so nothing needed to do there.

This will not 100% work on all VDB Providers or Embeddings Engines, since some comes with specific configurations - for instance Image Embeddings Engine from AWS. But it will work for most cases, so you can make a fairly generic recipe for AI Search.

Proposed resolution

  • Make it possible to set a default VDB provider, where you setup operation types.
  • Make a helper method in the plugin manager where VDB provider's can opt in to be a default VDB Provider (or maybe this is implied?)
  • Make a listener for uninstalling VDB Providers, that removes the default VDB Provider.
  • Make it possible to choose a default Embeddings Engine and default VDB Provider as database when setting up a Search API server of type VDB.
  • Make sure that it fetches embeddings dimensions when the Search API server config is being saved if the set_dimensions is set to false and the database is set to default.
  • Add a recipe for adding keys dynamically, so we also can setup a specific VDB Provider this way, see how AI Provider custom recipe does it. It could also be solved here ✨ Create custom recipe action for Key module Active , but its some more efforts, since this should cover all the ways you can save a key.

User interface changes

Changes to the Search API server configuration form, when using an VDB Provider. We add the default as the database (you still need to fill out database_name, collection, metric). We do set 0 as the automatic number on the ajax loading of dimensions.

API changes

No interface changes should be needed, it could be discussed if it makes sense to make a abstract Form for setting up the VDB Provider that can opt in to be default, but that will not have any breaking issues.

✨ Feature request
Status

Active

Version

1.0

Component

AI Search

Created by

🇩🇪Germany marcus_johansson

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

contrib.social Blog FAQ Discussions
Production build 0.71.5 2024