Contrib.social

Database connection timeout during parallel indexing

Open on Drupal.org →
Created on 25 June 2025, about 1 month ago

Setup

  • Solr version: 8.11.4
  • Drupal Core version: 10.3.14
  • Search API version: 1.38
  • Search API Solr version: 4.3.10
  • Configured Solr Connector: Standard

Issue

First of all, thank you for the parallel indexing feature. It's a game changer for indexing large amount of content.

The issue I run into is that the search-api-solr:index-parallel command can run into a database connection timeout (e.g. MySQL server has gone away) when child processes are running too long and their overall runtime (the elapsed time between the first process is started and the last one is finished) exceeds the database connection's idle timeout (e.g. wait_timeout).

In my special case wait_timeout is 3600 seconds and there is no problem when parallel indexing finishes within an hour. But when the amount of content is bigger and indexing takes more than an hour, a MySQL server has gone away error occurs at the very end of the indexing process. Even if all threads finished successfully otherwise. This is because when search-api-solr:index-parallel is waiting for the child processes, it does not interact with the database at all, leaving the connection idle. And when child processes are finished, the next function call $this->commandHelper->resetEmptyIndexState() indirectly triggers a database interaction and that's causing an error. This is a problem during deployments because the command's exit code is not 0.

Now, with all this in mind someone may say that "ok, then just increase the wait_timeout value" and/or "increase the number of threads, so the processes will be faster". Unfortunately, neither of these are generally good solutions because:

  • In a certain environment you may don't have an option at all to change the timeout setting but even if you do, values like 3600 are already too much and it's not secure to increase it further.
  • Above a certain level, increasing the number of threads won't speed up the overall process or the increase would have other side effects (e.g. higher load) that you want to avoid.

As a conclusion, it seems that the only generally good solution to this problem is that if the implementation ensures that the database connection is kept alive while the parent process is waiting for the child processes.

🐛 Bug report
Status

Active

Version

4.3

Component

Code

Created by

🇭🇺Hungary balazswmann

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

contrib.social Blog FAQ Discussions
Production build 0.71.5 2024