Queue confusion on replicated databases (auto_increment_offset)

Created on 14 November 2019, over 4 years ago
Updated 27 January 2024, 5 months ago

We've been seeing an issue on our production environment where cache tags are not getting cleared from the Varnish server using the Late Runtime Processor.

For example, when the tags to be cleared are as follows:
entity_types node:21604 node_list content_moderation_state:2339 content_moderation_state_list

on our production environment, it would reliably clear these tags instead:
entity_types content_moderation_state:2339 node_list content_moderation_state:2339 content_moderation_state_list

So the node:21604 tag is missing, and replaced by a duplicated instance of the content_moderation_state tag.

This environment is running on a Galera database cluster. One of the tricks necessary for multi-master MySQL replication is to change the behaviour of AUTO_INCREMENT:

From https://mariadb.com/kb/en/library/mariadb-galera-cluster-known-limitations/

Do not rely on auto-increment values to be sequential. Galera uses a mechanism based on autoincrement increment to produce unique non-conflicting sequences, so on every single node the sequence will have gaps.

In the method DatabaseQueue::createItemMultiple, there is an assumption that the IDs generated by the database will be sequential:

      // A multiple row-insert doesn't give back all the individual IDs, so
      // calculate them back by applying subtraction.
      for ($i = 1; $i <= count($records); $i++) {
        $item_ids[] = $id;
        $id++;
      }

Which causes the databases IDs recorded in the TxBuffer to be incorrect, so the wrong object gets loaded from the buffer in QueueService::claim, leading to the tag confusion issue.

πŸ› Bug report
Status

RTBC

Version

3.0

Component

Code

Created by

πŸ‡¬πŸ‡§United Kingdom grahamC Oxford, UK

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

Production build 0.69.0 2024