claimItem in the database and memory queues does not use expire correctly

Re-roll for 11.x

The last submitted patch, 65: 2893933-queue_lease_time-11.x-64.patch, failed testing. View results →

Adjusting 11.x tests in new CronSuspendQueueDelayTest.php

Reroll for 11.x looks good, my concerns from #63 about @see are addressed.

Setting RTBC.

Hopefully we'll be able to backport this to 10.0 and/or 9.5.

I want to give a special tangential shout-out to ✨ Refactor cron queue processing into its own class Needs work because I just spent a bunch of time trying to figure out a bespoke queue implementation that should have been an easy subclass of a generic queue handler. Tangent over.

+++ b/core/lib/Drupal/Core/Queue/DatabaseQueue.php
@@ -114,13 +114,19 @@ public function numberOfItems() {
-        $item = $this->connection->queryRange('SELECT [data], [created], [item_id] FROM {' . static::TABLE_NAME . '} q WHERE [expire] = 0 AND [name] = :name ORDER BY [created], [item_id] ASC', 0, 1, [':name' => $this->name])->fetchObject();
+        $item = $this->connection->queryRange('SELECT [data], [created], [item_id] FROM {' . static::TABLE_NAME . '} q WHERE (([expire] = 0) OR (:now > [expire])) AND [name] = :name ORDER BY [created], [item_id] ASC', 0, 1, [
+          ':name' => $this->name,
+          ':now' => \Drupal::time()->getCurrentTime(),
+        ])->fetchObject();

@@ -142,8 +148,7 @@ public function claimItem($lease_time = 30) {
-        ->condition('item_id', $item->item_id)
-        ->condition('expire', 0);
+        ->condition('item_id', $item->item_id);

Haven't made it all the way through this patch because I got hung up on this. I don't understand _why_ the change is needed. In addition if we read above the update the second block of code is part of we see the code is using the expire = 0 part of the condition to ensure only one process can claim an item. The current code will always succeed in updating removing this protection.

Doing some triage.

patch kill the issue, but need to find the core of error.

The 10.1.x patch from #62 doesn't apply to 10.2.x but the 11.x patch from #67 does, testing whether this still fixes the issue for me

Patch #67 seems to no longer apply cleanly to 10.3

This re-roll patch will cleanly apply the patch referenced in #67 to 10.3.x.
The main issue being QueueWorker being un-commented out in the two files listed:
core/modules/system/tests/modules/cron_queue_test/src/Plugin/QueueWorker/CronQueueTestDatabaseDelayException.php
and
core/modules/system/tests/modules/cron_queue_test/src/Plugin/QueueWorker/CronQueueTestLeaseTime.php

So the patch was looking for commented-out code where it has since been applied in 10.3.

Rerolled patch #75 for Drupal 10.3

Seems like a serious issue. I'd like to move this to the queue component because claimItem() bugs affect non-cron queues as well. Alas I dont think that component exists.

"bugs affect non-cron queues as well"
Do you have an example? I've not run into this issue and there's conceptual explanation but not real world examples.

Looks like the subsystem tag was added for me and I added the review so removing the tag.