Create a Milvus.io plugin for openai_embeddings module

Hey there,

I'd love for you to consider collaborating on existing modules first. I'm excited to share that I'll be releasing the code for ' Search API Milvus → ' on Drupal.org this weekend.

It's really important for us to build modules that can easily integrate with other ecosystems and modules. A great strategy is to separate the embedding functionality from the vector database implementation. This is something I've focused on in my projects, like ' Augmentor ✨ Add Augmentor Search API Processors Fixed ' and ' Search API Embeddings → ' on Drupal.org.

By focusing on flexibility and keeping different concerns separate, we can create something that benefits the entire Drupal community.

Leveraging search API makes a lot of sense to me. Search API name space for the module also makes sense. Separate modules means shared code and concepts across these 2 + pinecone is of course problematic so perhaps we can eg chat in slack along with Kevin (and others who might be interested) to work out a way to eg have this sub-module handle shared functionality and have your two provide the Search API functionality. Let's chat!

Hi @eleonel,

I have given your module https://www.drupal.org/project/search_api_milvus → a whirl. While it does provide a back-end, it is a fair ways off from actually working even after my fairly large merge request here https://www.drupal.org/project/search_api_milvus/issues/3403280 🐛 Clean up API client and Milvus Backend, getting them closer to working state Needs review

Ultimately I believe there is more work in place in https://www.drupal.org/project/search_api_ai → which is very light as it leverages this module as a dependency. It seems like a better way forward to me as it would be far easier to keep the module light + share code between Pinecone, Weviate, Milvus and other future vector storage engines compared to a fragmented set of modules.

For that reason, I am setting this back to 'Active' and will progress on it here.

In any case, thanks for raising - it is a valid point to consider reducing duplication; however, in this case my opinion is that its better to contribute the search_api_milvus code to search_api_ai and here particularly given the state of the code there.

Thanks,
Scott

Thanks, replied here: https://www.drupal.org/project/search_api_milvus/issues/3403280#comment-... 🐛 Clean up API client and Milvus Backend, getting them closer to working state Needs review

Thanks for the info, I'll follow that thread. For now continuing here. Mostly there with this https://git.drupalcode.org/issue/openai-3402579/-/tree/3402579-create-a-... and here https://www.drupal.org/project/search_api_ai/issues/3403577 📌 Implement Milvus as a vector storage backend Active

Will be creating some follow-ups, but working, and also works with https://www.drupal.org/project/search_api_ai/issues/3403577 📌 Implement Milvus as a vector storage backend Active

Automatically closed - issue fixed for 2 weeks with no activity.

@scott_euser Would you pls have a look at this issue 🐛 Exception: Source IDs to delete are required by Milvus in Drupal\openai_embeddings\Plugin\openai_embeddings\vector_client\Milvus->delete() (line 229 of Active ? I cannot get it working with a Milvus instance.