- Issue created by @joshhytr
- Merge request !143427862: Amend SearchApiPineconeBackend::search() logic to set result items with IDs... → (Open) created by joshhytr
- Status changed to Needs review
9 months ago 2:13pm 14 March 2024
When using the search function on SearchApiPineconeBackend, the result search item objects returned have use the ID of the chunked item, IE the ID in Pinecone.
This is not the ID that is tracked in the search_api index and the function Drupal\search_api\Entity\Index::loadItemsMultiple() does not transform them into an ID that can be loaded from a Drupal datasource.
Therefore, when Drupal\search_api\Entity\Index::loadItemsMultiple() is called on result items from SearchApiPineconeBackend::search(), it loads no items with the warning:
"Could not load the following items on index {index_name} ...".
Amend the SearchApiPineconeBackend::search() function to create one result item for every tracked item in the index. IE, combining all results from a single indexed Drupal entity that has been chunked into its fields in Pinecone, into one result item with the original Drupal entity's tracked ID.
The matches are ordered by 'score' descending to we should be able to take the first result of any given entity and know that is the most relevant result.
This allows search_api to load the items from the datasource.
Drupal\search_api\Item\Item::getId() will return the ID of the search_api tracked entity and not the ID of the item in Pinecone.
This seemed like the best place to solve the issue, however, it may impact how the SearchApiPineconeBackend is being used on other sites and the metadata content that is returned with the result item will only contain the content from the most relevant chunk/field.
If anyone has suggestions of how I could better solve this, or if I'm looking at the problem wrong, please let me know. I am currently considering extending the SearchApiPineconeBackend to not chunk entities into fields and therefore keep the tracked ID in Pinecone. This will allow delete by ID too.
Thanks!
Needs review
1.0
Code