Prepare openai_embeddings module for multiple plugins beyond pinecone

Created on 8 November 2023, 8 months ago
Updated 9 November 2023, 8 months ago

Problem/Motivation

Currently Pinecone is set up as a plugin, but its hard-coded in in the queue worker.
A follow-up could be to define the config within the plugin instead of in the settings form, but trying to keep the scope of this small enough for easy review & merge.

Steps to reproduce

Attempt to create another plugin like Weviate or eventually MySQL/Postgres

Proposed resolution

Untie it a bit so other plugins can be used

Remaining tasks

MR coming shortly

User interface changes

User must choose the plugin. Update hook maintains the status quo by selecting pinecone for existing sites.

API changes

None

Data model changes

Adds plugin selection for vector client choice

πŸ› Bug report
Status

Fixed

Version

1.0

Component

OpenAI Embeddings

Created by

πŸ‡¬πŸ‡§United Kingdom scott_euser

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @scott_euser
  • @scott_euser opened merge request.
  • πŸ‡ΊπŸ‡ΈUnited States kevinquillen

    Thanks. I had started some of this work with that in mind, configurable plugins, but I forget which part I got hung up on. Some of that remains in the module code (but unused).

  • Issue was unassigned.
  • Status changed to Needs review 8 months ago
  • πŸ‡¬πŸ‡§United Kingdom scott_euser

    Status quo is maintained with this, it just facilitates future issues here to add additional plugins that are more affordable compared to pinecone.

  • πŸ‡¬πŸ‡§United Kingdom scott_euser

    Thanks. I had started some of this work with that in mind, configurable plugins, but I forget which part I got hung up on. Some of that remains in the module code (but unused).

    Not a problem, thought I would pick it up from where you left off as I would like to try out Weaviate and I imagine future MySQL will let us add that as an option

  • πŸ‡ΊπŸ‡ΈUnited States kevinquillen

    This looks good. Made a few changes and resolved a merge conflict from the other namespace fix. I added some notes around namespace and deleteAll not working for Starter plans.

    One other thing, the 'Search test' and 'index stats' tab would need to pull a form from the configured plugin instead of what it is now. But that can be in a new ticket.

  • πŸ‡ΊπŸ‡ΈUnited States kevinquillen

    Found one other issue - the Pinecone plugin is not storing to or pulling from its plugin configuration. Likely because the schema is not defined. I added back the original config name to keep the queue worker functioning properly. That can be addressed in a new ticket as well. But things are functional as they were before.

  • Status changed to Fixed 8 months ago
  • πŸ‡ΊπŸ‡ΈUnited States kevinquillen
  • πŸ‡ΊπŸ‡ΈUnited States kevinquillen

    Added related issue: ✨ Finish VectorClient plugin code. Active

  • Automatically closed - issue fixed for 2 weeks with no activity.

Production build 0.69.0 2024