Parallel indexing using concurent drush processes

I was annoyed from the duration of indexing a complex site with a lot of custom PHP code that affects the indexing of each item. It took about 2 hours per run and I had to do it multiple times.
Unfortunately Search API is not prepared in any way to perform multiple indexing tasks in parallel. But my test system has enough CPU cores and Solr itself would be able to accept a much higher indexing load.
So I created a new drush command to run multiple index processes in parallel:
drush search-api-solr:index-parallel --threads=10
drush search-api-solr:index-parallel YOUR_INDEX_ID --threads=10

The approach is a bit hackish, but it works :slightly_smiling_face:
Using 10 “threads” the same indexing as before is done in 20 minutes! I think that this is a signifikant improvement :grin:

By default, different indexes get indexed in parallel using this new command.
But if you additionally change the tracker of an index to the new “Index parallel”, the different items of that index get indexed in parallel!

I think I’ll merge it into search_api_solr soon, but if you’re already interested:
https://github.com/mkalkbrenner/search_api_solr/commit/bdd44de65270d3e0e...

Automatically closed - issue fixed for 2 weeks with no activity.

Hi @mkalkbrenner, you might be interested in:

https://www.drupal.org/project/search_api/issues/3527726 ✨ Add a parallel tracker to allow concurrent indexing Needs review

which is a generic solution, which your parallel drush command could use as backend - without having to split the index into chunks.

Thanks for considering!