Database service not starting in some CI runs.

Created on 7 February 2024, 4 months ago
Updated 23 May 2024, 24 days ago

Problem/Motivation

We've been investigating for quite a while situations when the DB service is not available in core and contrib. It's hard to reproduce because re-running the jobs usually fixes the issue, but adding CI_DEBUG_SERVICES gives us extra debug information that can be useful

That was the case, and from @dww in https://www.drupal.org/project/gitlab_templates/issues/3414252#comment-1... β†’

Okay, here's a real failure from a job with CI_DEBUG_SERVICES enabled πŸŽ‰

https://git.drupalcode.org/project/address/-/jobs/752891

Logs are full of this:

[service:drupalci/mysql-5.7-database] 2024-02-05T21:59:53.851198940Z netcat: connect to localhost (127.0.0.1) port 3306 (tcp) failed: Connection refused
[service:drupalci/mysql-5.7-database] 2024-02-05T21:59:53.851225416Z netcat: connect to localhost (::1) port 3306 (tcp) failed: Cannot assign requested address

Here's the real culprit:

[service:drupalci/mysql-5.7-database] 2024-02-05T21:59:23.451301665Z 2024-02-05T21:59:23.451125Z 0 [ERROR] InnoDB: io_setup() failed with EAGAIN after 5 attempts.
[service:drupalci/mysql-5.7-database] 2024-02-05T21:59:23.451322612Z 2024-02-05T21:59:23.451154Z 0 [Note] InnoDB: You can disable Linux Native AIO by setting innodb_use_native_aio = 0 in my.cnf
[service:drupalci/mysql-5.7-database] 2024-02-05T21:59:23.451325887Z 2024-02-05T21:59:23.451266Z 0 [ERROR] InnoDB: Cannot initialize AIO sub-system
[service:drupalci/mysql-5.7-database] 2024-02-05T21:59:23.451328497Z 2024-02-05T21:59:23.451274Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
[service:drupalci/mysql-5.7-database] 2024-02-05T21:59:23.451331115Z 2024-02-05T21:59:23.451281Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
[service:drupalci/mysql-5.7-database] 2024-02-05T21:59:23.451333559Z 2024-02-05T21:59:23.451285Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
[service:drupalci/mysql-5.7-database] 2024-02-05T21:59:23.451338953Z 2024-02-05T21:59:23.451289Z 0 [ERROR] Failed to initialize builtin plugins.
[service:drupalci/mysql-5.7-database] 2024-02-05T21:59:23.451341452Z 2024-02-05T21:59:23.451292Z 0 [ERROR] Aborting
[service:drupalci/mysql-5.7-database] 2024-02-05T21:59:23.451344050Z 
[service:drupalci/mysql-5.7-database] 2024-02-05T21:59:23.451346775Z 2024-02-05T21:59:23.451304Z 0 [Note] Binlog end
[service:drupalci/mysql-5.7-database] 2024-02-05T21:59:23.451465428Z 2024-02-05T21:59:23.451358Z 0 [Note] Shutting down plugin 'CSV'
[service:drupalci/mysql-5.7-database] 2024-02-05T21:59:23.453778406Z 2024-02-05T21:59:23.453665Z 0 [Note] /usr/sbin/mysqld: Shutdown complete

However, it's not clear why that is happening, just from these logs. Wonder if there's other output being saved somewhere that might be useful. Hopefully @fjgarlin has a chance to review this and knows where to look for the underlying problem.

I followed up on slack here: https://drupal.slack.com/archives/CGKLP028K/p1707211277736559?thread_ts=...

It seems that the quickest workaround is to add some configuration to the my.cnf file. We might need a bigger and more robust fix somewhere else, but it's not clear where or what yet, so we should address the issue here if possible.

It seems to be happening on mysql-5.7 but I'd probably do it for the other mysql versions too.

Steps to reproduce

Can be duplicated on a local system.

for i in $(seq 1 15);
do
    docker run --rm --name resource_exhaust_$i drupalci/mysql-5.7:production  > /dev/null 2> /dev/null &
    sleep 30
done

Proposed resolution

Change my.cnf files for the images with a fix for that situation.

Remaining tasks

MR

User interface changes

API changes

Data model changes

πŸ“Œ Task
Status

Fixed

Component

PHP Containers

Created by

πŸ‡ͺπŸ‡ΈSpain fjgarlin

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

Production build 0.69.0 2024