Support ElasticSearch 8

Any update on this? To me it seems nodespark/des-connector should get an 8.x branch/version to make it work with ES8.

I opened https://github.com/nodespark/des-connector/issues/20 to get the ball rolling on 8.x. We are working against an EOL of ES 7 at this point per https://www.elastic.co/support/eol#elasticsearch. Once ES 9 is released, 7.x will no longer be supported.

8.0.x branch already removed nodespark/des-connector dependency and use ruflin/Elastica directly #3186932: Use ruflin/Elastica instead of nodespark/des-connector →

The Elastica folks are working on Elasticsearch 8.x support with good progress https://github.com/ruflin/Elastica/pull/2181, so when it landed we should create 8.x branch from 8.0.x depend on that new Elastica version.

However from what I see 8.x-7.x is diverged from 8.0.x, so before creating 8.x branch, changes from 8.x-7.x should be forward ported to 8.0.x.

I missed the commit https://git.drupalcode.org/project/elasticsearch_connector/-/commit/51d1.... Besides forward porting 7.x patches, what else is needed for 8.x support?

if you check this version of nodespark/des-connector https://github.com/nodespark/des-connector/blame/7.x/composer.json , the one that is required for 8.x-7.0alpha4, it is already also based on ruflin/Elastica.

I think there is some clarity missing in the project development. There is this 8.0.x branch that was an attempt to get rid of nodespark/des-connector in favour of ruflin/Elastica, but meanwhile 2 years ago nodespark/des-connector itself moved to ruflin/Elastica :)

mparker17 → changed the visibility of the branch 8.0.x to hidden.

I have been working on ElasticSearch 8.x support, using ruflin/Elastica as a backend library.

This work is far from complete (notably, nothing works yet and most of the tests fail!) but wanted to share my work thus far to show there is progress being made. The customer that I am working for is paying me to work on this issue, so I'm going to leave it assigned to myself while I work on it over the next little while. Thanks in advance for your patience!

Recap of what I've done thus far:

1. Mark compatibility with Drupal 10 in .info files.
2. Set minimum version for illuminate/support to 10.41.0 (since that's what I'm testing with).
3. Set minimum version for ruflin/elastica to 7.3 (since that's what I'm testing with).
4. Merged changes from the 8.x-7.x branch (because there were some changes made there since 8.0.x was forked from it)
5. Cleaned up the most obvious type errors in code and comments.
6. Optimized imports in all files, to remove references to things that are no longer used (and sort them by name, which phpcs is linting for now ✨ Add sniff to check and fix the order of Use statements Fixed ).
7. Fix two tests whose Overridden ::setUp() function wasn't conforming to the parent method signature.

With the advice of @sokru, my next steps will likely be:

1. look into 📌 Investigate search_api_opensearch as base for elasticsearch_connector Fixed (if we can build upon another module → , that would save a bunch of time and remove a bunch of code we don't need)
2. delete some unused D7 functions in the .module file, and functions marked as deprecated and scheduled for deletion in 8.0.x (less code to maintain, fix, etc.)
3. figure out why tests are failing and fix issues
4. look into making the Drupal.org GitLab CI → linters pass

I am using 8.x-7.x-alpha4 release with following patches:

      "drupal/elasticsearch_connector": {
        "[https://drupal.org/i/2977537] - Source Fields in Elasticsearch Index are arrays": "https://www.drupal.org/files/issues/2021-07-30/8.x-7.x-2977537-18-array-vs-scalar.patch",
        "[https://drupal.org/i/3102388] - Add Search API data source implementation": "https://www.drupal.org/files/issues/2024-01-17/3102388-elastic-search-document-16.patch",
        "[https://drupal.org/i/3183164] - Index time boost is deprecated": "https://www.drupal.org/files/issues/2023-09-28/index-time-boost-is-deprecated--3183164--7.patch"
      },

Elastic server 8.11.3. Indexing and querying works. Maybe some very specific things are not, but basics are working fine. I have setup with search_api.

Hey folks, sorry for not reporting my progress on this issue, and thus causing work duplication. I've also been juggling a bunch of other work at the same time.

I have not yet had a chance to look at @franadame's branch from this morning, but I will shortly. I'll unassign myself from this ticket in the meantime.

I've been working in the 3279558-support-elasticsearch-8 issue fork and on my local computer. On my end, I have successfully merged 8.x-7.x into the the 8.0.x branch, but I've been trying to get tests passing before I push my work to drupal.org. Once I have tests passing, I was planning to apply some of the patches from #11, add GitLab CI config → , and integrate a bunch of changes from 📌 Investigate search_api_opensearch as base for elasticsearch_connector Fixed .

Thanks again for your patience, and I apologize again for not communicating about my progress as well as I should have.

Follow-up to my last comment: I don't see any differences between the commits in my branch (3279558-support-elasticsearch-8) and the commits in @franadame's branch (3279558-elastic7-8, merge request !38).

@franadame, were you planning to contribute code to your branch, were you creating a merge request to get a unified diff you could test with, or something else? If you are planning to contribute code to this issue, perhaps we can collaborate?

franadame → closed merge request !38

Hi @mparker17
I just wanted to "move" this issue to be merged because ES8 is not currently working in D10.
I've just cheked your changes (when i created the MR) and currently is working on my project, so no contributions were needed apart from adapting my custom code to this new release in my project.

As an aside, I’ve submitted a pull request to add support for the Combined Fields query to the ruflin/Elastica PHP library: issue #2195, pull request !2196 for branch 8.x.

Most of the commits since Feb 15th have been trying to figure out how to spin up an Elasticsearch container to run tests against (it's been working locally since then, just not on drupal.org). I'm still working with Drupal.org GitLab CI folks in Slack to try to resolve this... sorry to add noise to this issue... I intend to squash all the commits once we have something working.

Hi everyone, I'm happy to report that the phpunit tests are finally passing in GitLab CI (but note there are a bunch of linter issues remaining, and we may still want to delete a lot of the cruft, as per 📌 Investigate search_api_opensearch as base for elasticsearch_connector Fixed )! Also, I've squashed all ~26 commits I had made to try to get GitLab CI working, so it's easier to review.

So, I think this is ready for some review, testing, and feedback now.

We need to improve documentation in the gitlab templates, but wanted to mention that you can easily skip lints if this is not desired with these variables https://git.drupalcode.org/project/gitlab_templates/-/blob/main/includes...

I spent weekend updating merge request on 📌 Investigate search_api_opensearch as base for elasticsearch_connector Fixed . I'd be much rather use the codebase from the search_api_opensearch unless there's a strong arguments to use MR's from this issue. Codebase on both issue introduce significant changes so people who have extended any classes/hooks from 8.x-7.x needs to update their codebase anyway. But I'm open to counter arguments, and if we'd choose to close this in favor of 📌 Investigate search_api_opensearch as base for elasticsearch_connector Fixed I'll transfer the credits to corresponding peoples.

Hi everyone, @sokru and I worked to add ElasticSearch 8 support to the 8.0.x branch in 📌 Investigate search_api_opensearch as base for elasticsearch_connector Fixed , using what we learned in this ticket, so I'd encourage you to try out the 8.0.x branch if you're looking for this feature.

Automatically closed - issue fixed for 2 weeks with no activity.