Add support for neural search (text embeddings)

Still work in progress, but the idexing side of things work.

Thanks Janez. Looks like a great start. Besides the linting errors, I can see there are still a lot of hard coded values, and we're missing tests. I know it's a complex area, but it would be good to have some basic docs on setup and links for further info.

Quick n dirty review for now as this is obviously still WIP - great to see movement in this area though!

slashrsm → changed the visibility of the branch query_side to hidden.

After watching Driesnote and looking into AI module → a bit I realized that we are basically re-implementing their provider plugins here. In order to avoid that I decided to depend on the AI module for providers. Updated MR assumes/uses ✨ Provide embeddings vector size Active , which add vector size function that we rely on.

I was looking into adding OpenSearch as a vector database provider and I found this issue by chance. I was wondering if it wouldn't be better to put this functionality in its own provider module? I started a project just for that before I found this issue by chance.

FYI the separate provider module has been deprecated in favor of this issue, though the current MR doesn't apply against the 3.x branch.

A recommended approach for vector indexing is an ingest pipeline. I wonder if this issue could be expanded to include support for that?

Started work on a more integrated approach. At this stage all the MR does is set index.knn = TRUE when creating the index.

In order to have knn enabled on an index, we need to set that option when creating the index. We can change it after.

This meant we needed to refactor the addIndex() method to not create then update settings, but to pass the settings at creation time. This refactoring could potentially be split out into a separate issue.

Ran into a bit of an issue with the pipelines. In order to have Opensearch generate the text embeddings, you need to specify text field to embedding field mappings when creating the pipeline. I don't think it would be easy to dynamically create a pipeline like this with search api.

I'm going to check out the https://www.drupal.org/project/ai_vdb_provider_opensearch → module to see if the built-in AI Search would work.

Updated the MR so choosing the Text Embedding (Vector) data type creates the field as a text field. Then the field mapper is decorated to create a paired knn_vector field. That will let us dynamically create the ingest pipeline.

Chatting with @mohit_aghera in slack and we have a new approach based on a semantic field type in OpenSearch 3.1. This greatly simplifies the amount of work needed to support neural search.

There is a bug with remote text embedding models, which was recently fixed and merged but not yet in a release. https://github.com/opensearch-project/neural-search/issues/1426

mstrelan → closed merge request !109

We have moved this to it's own project https://www.drupal.org/project/search_api_opensearch_semantic → with the latest from MR !107.

Please continue any work over there.

Marking this issue as Fixed to provide contribution credits.

kim.pepper → closed merge request !107

kim.pepper → closed merge request !64

Created 📌 Add a BeforeIndexCreateEvent Active for the new event.