At runtime track URLs that have already been queued and skip those.

I'm going to reopen this as I believe the original request has merit as a work around until a more clear solution is propose for 2851893.

While 2851893 focuses on duplicate queued items - there is a valid issue with the original report of duplicate queued items in the same request, as the buffer does not deduplicate items (noted in https://www.drupal.org/project/purge/issues/2851893#comment-15182977 ✨ Deduplicate Queued Items Active )

Importing config is one scenario we hit where memory limits got hit after importing a very large number of config changes.

Attached is a patch that does what was originally suggested - track the purged urls of the request in a similar way to how invalidated tags are tracked.

This deduplicates items in the buffer, but if you are concerned about duplicate items in the queue (from previous/subsequent requests) you will need additional solutions such as those suggested in 2851893.

I haven't added a test with this patch but can share my local test using a similar scenario as described in the issue summary.

Setup:

1. Added 6 blocks all set to show on all pages, exported config
2. Generated 30,000 items in the traffic registry using block list and the individual block tags
3. Change the weight (in config) for the 6 blocks
4. Import config with drush -vvv cim

Without the patch:

  [debug] Calling Drush\Drupal\Commands\config\ConfigImportCommands::doImport(Drupal\Core\Config\StorageComparer) [4.89 sec, 114.07 MB]
 [notice] Synchronized configuration: update block.block.block_1. [8.12 sec, 208.87 MB]
 [notice] Synchronized configuration: update block.block.block_2. [9.19 sec, 242.77 MB]
 [notice] Synchronized configuration: update block.block.block_3. [10.21 sec, 286.69 MB]
 [notice] Synchronized configuration: update block.block.block_4. [11.28 sec, 321.07 MB]
 [notice] Synchronized configuration: update block.block.block_5. [12.38 sec, 356.42 MB]
 [notice] Synchronized configuration: update block.block.block_6. [13.48 sec, 390.32 MB]
 [notice] Finalizing configuration synchronization. [14.41 sec, 395.7 MB]
 [success] The configuration was imported successfully. [14.41 sec, 395.77 MB]

Repeated with the patch:

 [debug] Calling Drush\Drupal\Commands\config\ConfigImportCommands::doImport(Drupal\Core\Config\StorageComparer) [5.07 sec, 114.64 MB]
 [notice] Synchronized configuration: update block.block.block_1. [7.67 sec, 172.27 MB]
 [notice] Synchronized configuration: update block.block.block_1. [8.38 sec, 172.52 MB]
 [notice] Synchronized configuration: update block.block.block_1. [9.03 sec, 172.68 MB]
 [notice] Synchronized configuration: update block.block.block_1. [9.71 sec, 173.42 MB]
 [notice] Synchronized configuration: update block.block.block_1. [10.38 sec, 175.12 MB]
 [notice] Synchronized configuration: update block.block.block_1. [11.04 sec, 175.39 MB]
 [notice] Finalizing configuration synchronization. [11.96 sec, 180.77 MB]
 [success] The configuration was imported successfully. [11.96 sec, 180.83 MB]

This is only for 6 blocks - so you can imagine the imapct for much larger deployment sets.

The patch looks simple and elegant , thanks Eric.

Works as expected.

Automatically closed - issue fixed for 2 weeks with no activity.