Add a bulk create method to the queue implementations

Created on 6 March 2019, about 6 years ago
Updated 14 May 2025, 3 days ago

Problem/Motivation

Current QueueInterface has means of creating items in the queues.
It currently supports only linear additions of items to the queue - one by one.

What is needed and supported in many queue servers and assumptions is that there should be a fast and efficient way to push multiple items for processing into the queue at once.

Proposed resolution

For example bulkCreateItems.
The idea is that this should allow for a more efficient data transfer of the items from the one that generates items and the server that is responsible for storing them. In core's case - SQL.

It's a safe addition, as it can be emulated if the backing server does not support it through a simple loop and fallback to the other method.

Remaining tasks

Discussion is this a D8 or D9 material.
Direction, patch, etc.

User interface changes

None - api addition: bulk creation of queue items.

API changes

New method: bulkCreateItems that accepts a list of items that are OK for use in the currently existing createitem method.

Data model changes

None.

Release notes snippet

TBD.

Feature request
Status

Active

Version

11.0 🔥

Component

base system

Created by

🇧🇬Bulgaria ndobromirov

Live updates comments and jobs are added and updated live.
  • Performance

    It affects performance. It is often combined with the Needs profiling tag.

Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • 🇦🇺Australia mstrelan

    To bike-shed here I would simply call it createItems or possibly createMultiple, similar to EntityInterface::loadMultiple.

    I'm not sure what is the BC way of adding methods to an interface. We could create a new interface that extends QueueInterface and provide a trait that has a default implementation with a simple loop.

    It looks like we could easily refactor \Drupal\Core\Queue\DatabaseQueue::doCreateItem to insert multiple items at once with a bit of massaging. In fact, the simple_sitemap module already does this in SimpleSitemapQueue. Not quite sure about the return value though.

    At a glance the redis client already allows preprending multiple items, so redis module can easily make use of that too.

    In core there are not a lot of places that could make use of this. I've identified the following:

    • _batch_populate_queue - multiple batch operations
    • locale_cron_fill_queue - same as above
    • \Drupal\media\Entity\Media::postSave - download thumbnails for multiple translations

    So we're probably only saving a handful of queries there. But there are more places in contrib that could also make use of this. A few I see in the project I'm currently working on:

    • entity_reference_revisions_entity_delete
    • \Drupal\linkchecker\LinkCheckerService::queueLinks
    • \Drupal\workbench_email\EventSubscriber\WorkbenchTransitionEventSubscriber::onContentModerationTransition

    I'm not sure that system.module is the right component though, so changing to base system.

Production build 0.71.5 2024