Queue lease time and queue timeout should be configurable separately

Created on 17 January 2024, 5 months ago

Problem/Motivation

I have heavyweight queue jobs that only need to run occasionally. Each item can take between 2 and 15 minutes to process. I run cron every minute and so workers build up in parallel to process these jobs.

If I set the lease time/timeout to 3 minutes (to avoid too much strain on the database from the jobs) then some work is repeated because the lease expires and another worker picks up the job again while the previous worker is still processing it.

If I set the lease time/timeout to 10 minutes then the site starts to fail in other ways because of load on the database.

Steps to reproduce

Proposed resolution

Allow lease time and queue timeout to be set separately, e.g. for the example above I would like job timeout of 3 minutes (so the worker exits after this point once an item is complete) but lease time of 15 minutes (so long running work is not repeated).

Remaining tasks

User interface changes

API changes

Data model changes

πŸ“Œ Task
Status

Active

Version

11.0 πŸ”₯

Component
CronΒ  β†’

Last updated 3 days ago

No maintainer
Created by

πŸ‡¬πŸ‡§United Kingdom longwave UK

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @longwave
  • πŸ‡¬πŸ‡§United Kingdom longwave UK

    Ultimate Cron does nothing special here - this is a core issue:

            $end = $this->time->getCurrentTime() + $info['cron']['time'];
            $queue = $this->queueFactory->get($queue_name);
            $lease_time = $info['cron']['time'];
            while ($this->time->getCurrentTime() < $end && ($item = $queue->claimItem($lease_time))) {
    

    I need two different values for the two different uses of $info['cron']['time'].

Production build 0.69.0 2024