Refactor Search API integration to pass more of the work to the VBD Provider

I added comments to the MR just to make it easier to review. Ultimately though its just moving things over from search api backend into the vdb providers to give the individual providers more control.

I see this added the "probots-io/pinecone-php": "^1.0" requirement.
This comes with a php 8.2 requirement that basically stops allowing us to use it on Drupal 10.2 where 8.1 is still supported.

Is this truly a hard requirement?

Does this mean ai search itself requires 8.1 even if you're not using pinecone? I think we need the ai module and ai search to support 8.1.

Yeah, as it's part of the root composer.json it means whenever you require drupal/ai you get all of the requirements and theirs.

Can imagine it's difficult with all the providers and their libraries and dependencies and maintenance of those.

I have no idea how to make dependencies requirements of specific sub-modules only. Thus far all dependencies per sub-module have been just tossed in the ai core composer.json like this. Open to suggestions.

Maybe some mechanism to have all these (like qdrant, pinecone, openai, anthropic) as a suggested composer dependencies (https://getcomposer.org/doc/04-schema.md#suggest) that block enabling the module if not installed. E.g. via https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21Extension...
It makes it harder for end-users to install sub-modules though, but maybe that's the lesser of two evils?

Can we move this to a separate issue though please rather than reviving unrelated closed issue?

Created here, https://www.drupal.org/project/ai/issues/3478274 ✨ Consider moving sub-module specific composer.json requirements into suggest rather than require Active - will update with notes

Automatically closed - issue fixed for 2 weeks with no activity.