Contrib.social

Queue table keeps growing on high-traffic site

Open on Drupal.org →
Created on 2 September 2022, about 2 years ago
Updated 12 April 2024, 8 months ago

Problem/Motivation

The queue is not keeping up with processing radioactivity incidents. With the database copied to the test site, and cron running only every hour the queue does go down, but only about 42 each cron run, so i presume the issue is that on the live site it is accumulating radioactivity incidents faster than they get processed in the queue.

I don't think we want to run cron much more often than what we currently are, every ten minutes, on the live site, so not sure what we can adjust to make the queue go through radioactivity_incidents faster?

We have 81,566 radioactivity_decay items in the queue table after truncating this morning— 142mb. It was more than 10GB before we truncated!

Steps to reproduce

Proposed resolution

Remaining tasks

  • Change cron message using entities instead of incidents, queue for processing instead of processed.
  • Add documentation about altering queue processing time
  • Add documentation about queue debugging

User interface changes

API changes

Data model changes

💬 Support request
Status

Active

Version

4.0

Component

Code

Created by

🇺🇸United States mlncn Minneapolis, MN, USA

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • Comment 8 months ago →
    🇮🇹Italy realgiucas

    I have the same problem as @mlncn and I want to thanks him for the investigation and testing work.
    I myself have done tests on the local ddev environment (rather powerful hw) and no more than 4 queue records per second are processed.
    On the production server (on staging site) it reachs 5 records per second. Absolutely not enough for a high traffic site.
    As a result, the queue table is constantly growing on the production site.
    Setting 600 seconds for the cron time of the radioactivity_incidents and radioactivity_decay queues would probably solve the problem but, I don't know why, only on the production site hook_queue_info_alter() is not executed when cron starts. Same enabled custom module code works on staging site. Maybe some different settings? Any suggestion?
    thx

contrib.social Blog FAQ Discussions
Production build 0.71.5 2024