Add support for queueing invalidations

Created on 8 November 2023, over 1 year ago

Problem/Motivation

When Drupal removes temporary files, the number of delete actions causes CloudFront errors due to the volume of individual invalidations being processed.

Steps to reproduce

  1. Modify or delete a number of files
  2. Wait 6 hours (or your site's determined amount of time between delete and removal of files
  3. View logs to see AWS Rate Limit errors

Proposed resolution

  • Allow the user to select between immediate or queued processing
  • Require the user to select the queuer to process invalidations
  • Adjust logic to queue invalidations instead of using selected process for immediate invalidation

Remaining tasks

  • Submit merge request with form and logic changes

User interface changes

  • Required select for Workflow: queued or immediate
  • Required select of Queuer to submit queued invalidations to
✨ Feature request
Status

Active

Version

1.1

Component

Code

Created by

πŸ‡ΊπŸ‡ΈUnited States kevinb623

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @kevinb623
  • πŸ‡¦πŸ‡ΊAustralia dpi Perth, Australia

    The motivation is good, however the implementation needs work

  • Status changed to Needs work 10 months ago
  • πŸ‡¦πŸ‡ΊAustralia dpi Perth, Australia
  • First commit to issue fork.
  • πŸ‡ͺπŸ‡ΈSpain omarlopesino

    This issue is really interesting, thanks for the contribution @kevinb623 . Supporting queues may drastically improve the performance when there are too many URLs to purge.

    Also many thanks for helping reviewing @dpi, it saved me time.

    The most priority changes needed to merge this are:

    1. Make form labels translatable
    2. Hook install to configure existing sites.
    3. Coding standards must pass, Currently, this module is integrated with Gitlab CI to help on this purpose.
    4. Ideally, applying every suggestion mentioned in the feedback.
  • πŸ‡ͺπŸ‡ΈSpain jonhattan Plasencia

    I thing this is going the wrong direction. Purge API is not being used the right way by the module and thus, by this MR.

    Some things I've seen so far:

    With respect to the general module implementation

    1. The user should not need to choose a processor

    Choosing a processor is useless. Despite the processor, it will purge files on the fly when invoking the hook in the middle of the request (not on cron, not on late runtime). Purge's invalidate() requires a processor indeed, but it is not used at all. See https://git.drupalcode.org/project/purge/-/commit/a3632d03ee72b0a09aa1b2...

    Instead of processing invalidations on the fly, it should rely on a queue and the system-wide configured processor. Otherwise, on any failure, we're exposed to lose pending invalidations since they're not tracked.

    If immediate purge were necessary for some reason, it could be opt-in in the module configuration. This should show a warning message, and not rely on the user choosing a processor, but passing invalidate() any of the enabled processors since it is a required parameter although useless.

    2. The module should warn and inhibit, if no url (or wildcardurl) invalidator processor is available.

    It passes silently if no compatible invalidator is enabled.

    With respect to this MR

    1. Purge queue is mandatory in purge configuration.

    No need to choose a queue.

    2. To enqueue items implement a PurgeQueuer plugin and use @purge.queue service

    There's an implementation easy to follow in this module: https://git.drupalcode.org/project/url_purge_aggregator/-/tree/1.x/. See https://git.drupalcode.org/project/url_purge_aggregator/-/blob/1.x/url_p...

Production build 0.71.5 2024