Document doesn't have a `search_api_document_id` attribute after upgrading 1.0.0 to 1.1.0-rc1

Hmm this is very weird, it looks as if you were updating from this 🐛 Remove the possibility to add field with machine name id Fixed dev version, but this functionality was reworked/removed before 1.1.0-rc1. That issue is the only instance of search_api_document_id that I know of.

Did a full DB and site files clone to a dev instance.

Restored to 1.1.0-rc1.
The second site works flawlessly.

The first one breaks again with the error in the title Document doesn't have a `search_api_document_id` attribute.

If I index with the older SearchApiMeiliSearchBackend.php and then switch to the current version, at least I have a version that somehow works for both sites. Indexing new (special?) nodes however will not work in this setup I guess.

Something's off.

The error
Drupal\search_api\SearchApiException while adding Views handlers for field on index mydomain Node Index: Could not retrieve data definition for field '' on index 'mydomain Node Index'. in Drupal\search_api\Item\Field->getDataDefinition() (line 482 of /var/www/drupal/web/modules/contrib/search_api/src/Item/Field.php).
is also back again.

I'm pretty convinced at this point that is a bug indeed and not a configuration/environment problem on my end.

Finally found it!

Due to some previous version, I had a primary key search_api_document_id in the first page's index.
Dropped all items from the index and changed the primary key with
curl -X PATCH 'http://localhost:7700/indexes/myindex' -H 'Content-Type: application/json' --data-binary '{ "primaryKey": "id" }' -H "Authorization: Bearer mypass" directly in Meilisearch.

I am unsure whether this warrants some error handling to be included somewhere around createField() in src/Plugin/search_api/backend/SearchApiMeiliSearchBackend.php
This seems to be the cause for the successive error stating field '', i.e. a field that never got created.

400   protected function getSpecialFields(IndexInterface $index, ItemInterface $item = NULL): array {
401     $fields = parent::getSpecialFields($index, $item);
402     $fields['id'] = $this->getFieldsHelper()->createField($index, 'id', [
403       'type' => 'string',
404       'original type' => 'string',
405     ]);
406
407     if ($item) {
408       $fields['id']->setValues([MeilisearchUtils::formatAsDocumentId($item->getId())]);
409     }
410
411     return $fields;
412   }

createField() comes from Search API's src/Utility/FieldsHelper.php, which has no error handling either. It in turn calls new Field() which I was unable to track down further.

Why exactly I got the error only for two nodes is still a mystery to me.

Unsure what to do with this ticket now. This will most likely never hit anyone.
Please decide whether some of the errors presented should get some better handling and/or tests.

Our posts overlapped. Yes, your diagnosis is absolutely correct!

I don't think that this issue needs extra handling since it happened between dev versions of the module, those are usually not recommended for production use. I'll leave the issue open so maintainers can decide what should be done. However, one thing I do think should probably be changed, is to add an explicit definition of what field should be considered as ID on meilisearch index creation, so it doesn't have to choose the field automatically.

Didn't I fix this in 🐛 Primary key inference fails for Meilisearch >=1.x and fields ending in "id" (includes patch) Fixed already? This is part of the current 1.1.0 release (and I guess in 2.0 as well, did not have the time to check, yet) at least. It's hard coded to id.

Directly handling the problem at hand does not really make sense, yes.
I just think the error handling is not ideal, this might be improved.

I could imagine situations (that was part of the discussion in some other issue as well) where an existing index might get attached to a setup running this module. This could pretty easily hit the error described here, if the primary key differs. And I think it should get caught cleanly.

Ahh yeah missed that, I was looking at doing this at index creation so in the public function addIndex(IndexInterface $index) method adding primaryKey in the options is also a way to set the primary key.

Yeah, the error handling could be improved. It would be much easier if meilisearch had any way to check what was set as ID. This way a simple fix would be to use whatever was preset by the index as ID, have to look more in to this.

It does have a way!
If you request the index' details with
xidel -s -e '$json' -H 'Content-Type: application/json' -H "Authorization: Bearer mypass" 'http://127.0.0.1:7700/indexes/mydomain'
(might as well use curl, I use xidel because it understands and formats JSON and XML, can use xquery/xpath, etc.)
you get some fields back, one of them being the primary key:

{
  "uid": "mydomain",
  "createdAt": "2023-07-12T13:48:59.640002336Z",
  "updatedAt": "2023-08-24T12:03:38.907236139Z",
  "primaryKey": "id"
}

I am however not sure whether meilisearch_php exposes that, but gathering from https://php-sdk.meilisearch.com/classes/Meilisearch-Endpoints-Indexes.html it seems so.

So looking at how search api works, this becomes a bit more complex. We'd have to create a custom datasource, where the admin could then map a custom ID. Now to be honest we'll have to eventually add datasource as a new feature, especially if we want to support data created by some service outside Drupal.

I'd say we should restrict the scope here to whether error handling makes sense, regardless of how/where errors originate, to potentially give a better clue for admins/devs if they occur.

Having differing data sources sounds like a useful addition that will have its application but should probably be created as a separate feature.

The problem comes down where to do this error handling. Looking at that specific error ... Could not retrieve data definition for field '' on index ..., this happenes on View handler, when it trys to map a field from meilisearch with Drupal storage data and not in any code we wrote. The weird part is where does it get that filed with empty key, I would have understood if error happened on 'id' or 'search_api_document_id'

We switched back to using id so existing indexes still work correctly. The only times this could happen is either when using development branch and some code changes between versions, if someone modifies the index by hand (cURL, xidel, etc.) or some 3rd party service. In this last case that datasource feature would help admins map fields correctly.

So the Could not retrieve data definition error is potentially caused by the Search API module itself 🐛 PHP 8.1 Deprecated Function Needs work .

@admirlju Do we still need to do something here? This issue as you mentioned would only occur if reindexing happened while on the dev branch in between the issues we were fixing regarding the document id and @tgoeg probably did the reindexing between the issues. If I remember correctly we added a BC which had to be reverted.

I'm closing this issue for now, feel free to reopen if I missed something.