only the paid version of pinecone works with this (dependency on namespaces)

Created on 30 October 2023, 8 months ago
Updated 4 January 2024, 6 months ago

Problem/Motivation

When you install this module and want it to work with pinecone, you need to take the paid version of pinecone.
The free version of the API does not support Namespaces (and deleteAll)
If I manually uncomment all the namespaces and the deleteAll then I can get it to work with the free pinecone API.

Steps to reproduce

Install the openai_embeddings module
create an FREE index on pinecone
Indexing will fail.
create a paid index on pinecone
indexing will succeed

I commented out the "namespace" properties in the api calls and the "deleteAll" and then indexing works on the Free index.
The data does not seem to delete (did not check yet in detail).

Proposed resolution

Maybe add a check in the pinecone configuration "Free index (limited to X nodes) "
which results in no namepaces added to the api calls and the removal of the deleteAll call?

Or just documenting that the integration only works with the paid (70 EUR/month) version of pinecone.

Remaining tasks

User interface changes

API changes

Data model changes

✨ Feature request
Status

Fixed

Version

1.0

Component

OpenAI Embeddings

Created by

πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @wouters_f
  • πŸ‡ΊπŸ‡ΈUnited States kevinquillen

    When did this change? Most of my experimentation with Pinecone was done with the free version.

  • πŸ‡¨πŸ‡¦Canada PierreMarcel Toronto, Canada

    Same issue here, based on some of the initial documentation and earlier videos it was possible to run on the free Pinecone tier.

    GuzzleHttp\Exception\ClientException: Client error: `POST https://drupal10-d6bf3b3.svc.gcp-starter.pinecone.io/vectors/delete` resulted in a `400 Bad Request` response: {"code":3,"message":"The requested feature 'Namespaces' is not supported by the current index type 'Starter'.","details" (truncated...) in GuzzleHttp\Exception\RequestException::create() (line 113 of /app/drupal/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php).

  • πŸ‡ΊπŸ‡ΈUnited States kevinquillen

    Well thats a bummer. Pinecone and its peers are in trouble with the advancements in OpenAI offering and MySQL/Postgres/Solr etc implementing vector storage and retrieval. But that is another story. I will look into this.

  • πŸ‡ΊπŸ‡ΈUnited States kevinquillen

    Unfortunately it seems like reworking the code to support free and not free could be a bit of work to cover all the areas, as namespaces used to be required. Is namespace only restricted under querying, or is it not available whatsoever on gcp-starter? If you can't delete records under the free tier anymore that almost seems like a non-starter because that would taint your future vector searches. Is the service really $75 a month?

    Maybe adding self identifying config (what subscription tier are you) and then checking that config value in code would be a stop gap, but it might be a bit before I can get to that. Patches welcome.

  • πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

    I think if you leave the namespace empty (and a message: leave empty for no namespace) you could refrain from sending it.
    It seems not allowed on any api call (on first sight, not checked on all of them).
    The 70 (it was more I think) euro's is what it said on the pricing pages.
    But I see that that already changed to some sort of pay as you go pricing.

  • First commit to issue fork.
  • @scott_euser opened merge request.
  • Status changed to Needs review 8 months ago
  • πŸ‡¬πŸ‡§United Kingdom scott_euser

    This makes it work at least - I added a warning about what I understand to be the impact:

    • Update hook ensures new disable option is not enabled for existing integrations
    • Disable option available in pinecone settings with a warning
    • Disabling stops the queue worker from setting a namespace
    • Pinecone plugin only adds the namespace arg if its set

    Thanks!

  • πŸ‡ΊπŸ‡ΈUnited States kevinquillen

    LGTM. Thank you!

  • Status changed to Fixed 8 months ago
  • πŸ‡ΊπŸ‡ΈUnited States kevinquillen
  • πŸ‡¬πŸ‡§United Kingdom scott_euser

    Thanks!

  • πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

  • πŸ‡§πŸ‡ͺBelgium wouters_f Leuven

    nice. Thanks!

  • Automatically closed - issue fixed for 2 weeks with no activity.

  • Status changed to Fixed 6 months ago
  • πŸ‡¬πŸ‡§United Kingdom scott_euser

    The starter plan now actually supports namespaces, so will need to get this reverted:
    https://docs.pinecone.io/docs/release-notes#december-2023

Production build 0.69.0 2024