API for handling unique queue items

namespace Drupal\my_class\Queue;

use Drupal\Core\Queue\DatabaseQueue;

/**
 * Custom queue implementation to merge on creation, preventing duplicates.
 *
 * @ingroup queue
 */
class MergeDatabaseQueue extends DatabaseQueue {

  /**
   * {@inheritdoc}
   */
  protected function doCreateItem($data): ?int {
    // This is copied and modified from DatabaseQueue::doCreateItem().
    // Merge instead of insert to prevent duplicates. An item is updated if it
    // already exists.
    $serialized = serialize($data);
    $query = $this->connection->merge(static::TABLE_NAME)
      ->keys([
        'name' => $this->name,
        'data' => $serialized,
      ])
      ->fields([
        'name' => $this->name,
        'data' => $serialized,
        // Note that this will update the 'created' field for an existing item.
        // We cannot rely on REQUEST_TIME because many items might be created
        // by a single request which takes longer than 1 second.
        'created' => \Drupal::time()->getCurrentTime(),
      ]);
    return $query->execute();
  }

}

the queue_unique contrib module already handles this - it ignores a new queue item if there is an existing one with the same data (or really the same sha256 of the serialized data)

https://www.drupal.org/project/queue_unique →

It is simple module, but as now, it lack maintenance, so perhaps adopting it to core would be good thing? The uniqueness support should be useful in core. We currently we roll custom code making sure we do not queue identical items coming from an external RabbitMQ system.

I agree. This functionality should be in Core.

I'll see if they will add me as a maintainer there, but it's simple enough we could likely merge into core

ok, also sounds like moving queue_unique into core is something that could be accepted, so we just need to make a MR here

I can't remember exactly what we ended up doing for update module, but pretty sure nothing much changed since 2012 so we could probably still convert it to this API once it's available.