Queue processors are leaking memory

Created on 6 March 2019, over 6 years ago
Updated 13 May 2025, 3 months ago

Problem/Motivation

I've added a custom queue and pushed 100K items in there.
There is a custom queue processor plugin that does:

public function processItem($item) {
  // Nothing...
}

The goal was to have as memory neutral system as possible, so the system can be left running for long periods of time and process queue items for as long there are items in the queue...

My problem comes from the fact that the whole thing started around 20-30 MB of ram usage that is kind of expected with Drupal. It started gradually growing little by little and it finished using data in the range of 90-100 MB, doing nothing...

Proposed resolution

TBD.

Remaining tasks

Find root cause.
Discussion...
FIx.

User interface changes

None.

API changes

None (most likely).

Data model changes

None (most likely).

Release notes snippet

TBD.

πŸ› Bug report
Status

Postponed: needs info

Version

11.0 πŸ”₯

Component

base system

Created by

πŸ‡§πŸ‡¬Bulgaria ndobromirov

Live updates comments and jobs are added and updated live.
  • Performance

    It affects performance. It is often combined with the Needs profiling tag.

Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • πŸ‡¦πŸ‡ΊAustralia mstrelan

    Is this still applicable? If so, please provide clear steps to reproduce. I don't think it's related to system.module, the queue system is in core/lib/Drupal/Queue. I couldn't find a "queue system" component so I guess base system will have to do.

  • πŸ‡¨πŸ‡¦Canada Charlie ChX Negyesi 🍁Canada

    There's nothing to do. While leaking memory in long running PHP processes is, alas, all too frequent there is no need to be long running, the queue is perfectly fine to run one process for each item -- possible even in an parallel.

  • πŸ‡¦πŸ‡ΊAustralia mstrelan

    Probably something like Symfony Messenger + Drupal: Realtime Queues and Cron β†’ is better for this anyway.

  • πŸ‡ͺπŸ‡ΈSpain fjgarlin

    You can have the queue job running every 10 minutes (for example) and use the --time-limit=590 parameter for a new clean run. That way the queue will be continuously running but should not take too much memory. Change 10 minutes for whatever your memory threshold / time is.

Production build 0.71.5 2024