Add delay to queue suspend

Created on 17 February 2021, almost 4 years ago
Updated 18 March 2023, almost 2 years ago

Problem/Motivation

Currently when a queue is suspended with SuspendQueueException in cron, the queue is skipped entirely and the next queue (if any) is processed. The queue is not revisited until the next cron run.

In some cases a worker may be able to determine when the queue can resume processing. It would be helpful for the cron queue runner to revisit the queue when its ready to process again.

Drupal already has \Drupal\Core\Queue\DelayedRequeueException which can currently specify how long to wait, however this is on the individual queue item level. Its not efficient to process all queue items and throw DelayedRequeueException for each.

  • A worker should be able to specify how long in seconds, and less than seconds, when to retry the queue.
  • We should keep the existing behavior where you can resume a queue before it's ready: queues should remain stateless.
  • A suspended queue should be non blocking: continue to process other queues while a queue is suspended.
  • If there are no queues ready to be processed, then sleep until the queue is ready. Except if the queue wont be ready in a timely fashion, see next.
  • A configurable timeout: in most cases we want the cron runner to exit in a timely fashion. If the queue wont be ready for a while, then skip the queue and dont resume until the next cron run.
  • API for suspending a queue for a time should be optional (opt-in), and be inspired by DelayedRequeueException::$delay . Ideally the delay should be specifiable in units less than whole seconds (float). This is helpful when dealing with external API's with per-second limitations (i.e Google).

Remaining tasks

None.

User interface changes

None.

API/Behavior changes

After each cron attempts one run per queue, if any queue threw exceptions, then SuspendQueueException cron will wait an amount of time indicated by the exception. Time is limited by queue_config:suspendMaximumWait service parameter. If the seconds limit is not exceeded then the queue will be retried, otherwise it will be skipped.

Data model changes

New container parameter.

Release notes snippet

Cron is now capable of revisiting a queue in the same run if a queue item when an SuspendQueueException is thrown with a sufficiently low suspension duration.

✨ Feature request
Status

Fixed

Version

10.1 ✨

Component
CronΒ  β†’

Last updated 19 days ago

No maintainer
Created by

πŸ‡¦πŸ‡ΊAustralia dpi Perth, Australia

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

Production build 0.71.5 2024