Make Vector Search Method Configurable?

Created on 2 July 2024, 8 months ago
Updated 31 July 2024, 7 months ago

Problem/Motivation

Search API AI had Cosine similarity as the only method of doing vector searches. Does this need to be pluggable?

Proposed resolution

There are alternatives such as dot product. I have heard:

- That you use use the same method for search as you did for creating the vectors.
- That you should use the same method for search the AI provider did when they created the model (I've also heard this isn't true).

Either way we may want to explore ways to make this configurable is there is any good reason to and make sure we do it properly (Such as we know which is best with which embedding model, or its tied to the configuration when first indexed).

https://medium.com/@serkan_ozal/vector-similarity-search-53ed42b951d9#:~:text=Cosine%20Similarity%20is%20a%20widely,in%20a%20multi%2Ddimensional%20space.

Feature request
Status

Fixed

Version

1.0

Component

AI Search

Created by

🇬🇧United Kingdom yautja_cetanu

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Production build 0.71.5 2024