Fix MemoryBackend::clearByPrefix with multiple items, and improve coverage and clarity

Created on 19 September 2017, about 7 years ago
Updated 7 April 2023, over 1 year ago

πŸ“Œ Improve test coverage of the flood memory backend test and convert it to a unit test Fixed was already merged, which fixed the initial motivation of this issue of resolving items added to memory flood within the same second.

The issue now is about...

Problem/motivation

  • Fix clearByPrefix where multiple items match.
  • Merging in the improved coverage from this issue.
  • Injecting time, so we can simulate the clock without needing to rely on PHP sleep calls. This is a CI performance optimization.
  • Improves English and verbosity of documentation.
  • πŸ“Œ Improve test coverage of the flood memory backend test and convert it to a unit test Fixed added unit tests for MemoryBackend, but is missing coverage for various scenarios such as clearByPrefix and with/without explicit $identifier parameter

Original issue summary

Although we use micro time as the key of events to guarantee uniqueness, but it is cast to an integer value:

  /**
   * {@inheritdoc}
   */
  public function register($name, $window = 3600, $identifier = NULL) {
    if (!isset($identifier)) {
      $identifier = $this->requestStack->getCurrentRequest()->getClientIp();
    }
    // We can't use REQUEST_TIME here, because that would not guarantee
    // uniqueness. (But this does not work !)
    $time = microtime(TRUE);

    // The key for example 1505837694.4603 is cast to 1505841294, the micro time cannot guarantee uniqueness.
    $this->events[$name][$identifier][$time + $window] = $time;
  }

We need a string key instead of the float key.

πŸ› Bug report
Status

Needs work

Version

10.1 ✨

Component
BaseΒ  β†’

Last updated about 16 hours ago

Created by

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • The Needs Review Queue Bot β†’ tested this issue. It either no longer applies to Drupal core, or fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

    Apart from a re-roll or rebase, this issue may need more work to address feedback in the issue or MR comments. To progress an issue, incorporate this feedback as part of the process of updating the issue. This helps other contributors to know what is outstanding.

    Consult the Drupal Contributor Guide β†’ to find step-by-step guides for working with issues.

  • @acbramley opened merge request.
  • Status changed to Needs review over 1 year ago
  • πŸ‡¦πŸ‡ΊAustralia acbramley

    I've rebased this onto a new MR against 10.1.x

    The problem is now that πŸ“Œ Improve test coverage of the flood memory backend test and convert it to a unit test Fixed also added unit tests for the MemoryBackend and it's hard for me to figure out what needs to be ported from the new unit tests this issue was adding so I've removed them all for now

    @dpi are you able to reconcile the tests?

  • Status changed to Needs work over 1 year ago
  • πŸ‡ΊπŸ‡ΈUnited States dww

    Thanks for all the work in here!

    Given that this issue is no longer about solving a major bug that potentially leads to data loss, but is now more of a cleanup task to add better test coverage, it needs:

    1. A new, more accurate title.
    2. A real issue summary, explaining what we're now trying to accomplish here.
    3. Probably needs a change record for the constructor changes to core/lib/Drupal/Core/Flood/MemoryBackend.php
    4. Probably needs a deprecation test (?)
    5. I opened a couple of minor MR comments, too.

    Isn't this now a normal task, not a major bug? If I'm wrong, please move it back, but re-classifying for now. But since I found it as a bug, tagging to be smashed. ;)

    Thanks again,
    -Derek

  • Assigned to dpi
  • πŸ‡¦πŸ‡ΊAustralia dpi Perth, Australia

    In progress.

  • πŸ‡¦πŸ‡ΊAustralia dpi Perth, Australia

    New summary and title

  • πŸ‡¦πŸ‡ΊAustralia dpi Perth, Australia

    Re #38

    no longer about solving a major bug [...] now more of a cleanup task

    Isn't this now a normal task, not a major bug?

    I encountered a bug during creation of unit tests for the recently added clearByPrefix, so reclassifying back to bug. Noting that expanding and clarifying coverage is the primary motivation for this issue

    A new, more accurate title.

    Done.

    A real issue summary, explaining what we're now trying to accomplish here.

    Done.

    Probably needs a change record for the constructor changes

    There's already a CR, and linked from code via trigger_error deprecation.

    Probably needs a deprecation test (?)

    Not sure about that.

    I opened a couple of minor MR comments, too.

    Resolved updating D9->D10 references and grammar issue.

    ---

    Tags

    • Removing Needs tests, this is about test improvement.
    • Removing Needs change record, already has CR
    • Removing Needs issue summary update, updated per previous comment
    • Removing Needs title update, updated per previous comment
  • Issue was unassigned.
  • Status changed to Needs review over 1 year ago
  • πŸ‡¦πŸ‡ΊAustralia dpi Perth, Australia

    Summarising new and existing code changes, especially since πŸ“Œ Improve test coverage of the flood memory backend test and convert it to a unit test Fixed and ✨ Add a clearByPrefix() method to the flood API Fixed :

    • Make MemoryBackend tests clearer to read and improve coverage, inject datetime for overridability and testability (no longer need sleep)
    • The English of comments and unit test method names committed with πŸ“Œ Improve test coverage of the flood memory backend test and convert it to a unit test Fixed is a little whack, so parts have been modified.
    • Fixed a couple of ints passed passed to strings, and strings passed to int with $threshold and $identifier
    • Improved code with [PHPStan] array shapes
    • testGarbageCollection replaces testNotExpiring
    • Note on change introduced for testNotExpiring:the changes in this PR where this comment is found "Progress time before window, event still exists after garbage collection."
      • It is fundamentally testing that garbage collection will not clear events as current time has not exceeded the expiry.
      • The existing check in testNotExpiring with comment ""Run cron", which clears the flood data and verify event is not allowed." is equivalent to the changes in this PR where this comment is found "Progress time before window, event still exists after garbage collection."
    • Added unit test coverage for the new prefix clear functionality added in Jan 2023 in ✨ Add a clearByPrefix() method to the flood API Fixed . Presently only Kernel tests, which πŸ“Œ Improve test coverage of the flood memory backend test and convert it to a unit test Fixed already resolved to move Kernel to Unit.
      • This also uncovered a bug with \Drupal\Core\Flood\MemoryBackend::clearByPrefix only clearing the first event.

    ---

    Back to NR

  • πŸ‡¦πŸ‡ΊAustralia dpi Perth, Australia
  • πŸ‡¬πŸ‡§United Kingdom catch

    This is flood rather than cache, since we don't have a component for flood, moving to 'base'.

  • Status changed to Needs work over 1 year ago
  • πŸ‡ΊπŸ‡ΈUnited States smustgrave

    Took a look (correct me if I'm wrong) but there are 2 open threads on the MR that appear to request some tweaks.

  • Update of 2910000-mr-1451-d93 β†’ patch so it can be applied for Drupal 9.5.x in the meantime.

Production build 0.71.5 2024