Contrib.social

Make batch a wrapper around the queue system

Open on Drupal.org →
Created on 28 September 2012, about 12 years ago
Updated 8 May 2023, over 1 year ago

I think a lot of people will agree that the Batch API, as it currently stands, is less than developer friendly. Let's rewrite it to be a wrapper around the Queue system in core.

chx said this on IRC:

[23:24:08] cweagans: batch needs to become a queue runner and reporter
[23:24:55] cweagans: *however* we need to be superb careful there
[23:25:08] cweagans: need to look at the reliablequeue semantics
[23:25:26] cweagans: perhaps even create a separte class for it even if it ships as an empty class extending the default queue
[23:25:29] cweagans: because the problem
[23:25:45] cweagans: you queue a repeating operating you do not know how many ops it will be , right?
[23:26:14] cweagans: so the worker works on it and if it needs more work, re-queues it... except if the worker dies then your batch just disappeared!
[23:26:44] cweagans: you need a queue which hands out the job and only the job when instructed and can reset the state for further work
[23:26:51] cweagans: ie. Amazon SQS is utterly unusable for this.
[23:27:05] cweagans: the default SQL is workable
[23:29:23] cweagans: http://privatepaste.com/8353271668
[23:29:32] cweagans: that's straight from beanstalk docs
[23:29:42] cweagans: beanstalkd is perfect too

I'm going to think about this for a couple of days, and then maybe come back with a proposal on how to improve the API. Opening this issue just to make sure it's tracked.

📌 Task
Status

Needs work

Version

10.1 ✨

Component
Batch  →

Last updated 4 days ago

Created by

🇺🇸United States cweagans Boise, ID, USA

Live updates comments and jobs are added and updated live.
  • Needs architectural review

    The issue is available for high level reviews only. If there is a patch or MR it is not to be set to 'Needs work' for coding standards, minor or nit-pick changes.

  • Needs issue summary update

    Issue summaries save everyone time if they are kept up-to-date. See Update issue summary task instructions.

  • Needs reroll

    The patch will have to be re-rolled with new suggestions/changes described in the comments in the issue.

Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • Comment almost 2 years ago →
    🇫🇷France fgm Paris, France

    Re: [23:26:51] cweagans: ie. Amazon SQS is utterly unusable for this.

    SQS is actually usable: if you don't call DeleteMessage, for example, after the VisibilityTimeout expires, the message becomes visible again and can be take by another worker. Alternatively, if you need a longer delay than the max of VisibilityTimeout, you can still call DeleteMessage and produce it back with another CreateMessage call, copying the origin metadata from the received message to the newly produced one.

  • Issue was unassigned.
  • Comment over 1 year ago →
    🇳🇴Norway steinmb

    Is this issue still valid?

contrib.social Blog FAQ Discussions
Production build 0.71.5 2024