Remove the queue option from the rebuild entity usage command

Created on 30 January 2025, 3 months ago

Problem/Motivation

Once πŸ“Œ Massively improve batch performance by reducing the number of queries Active , πŸ“Œ Allow users to clear the bulk inserting state flag on the UI Active and πŸ“Œ Stop loading entities when we do not need to Active we should remove the queue option as it can not be made to perform optimally as the batch and maintaining the code takes time. Also I think it might be untested.

Proposed resolution

Remove \Drupal\entity_usage\EntityUsageQueueBatchManager and anything that uses it.

Remaining tasks

User interface changes

API changes

Data model changes

πŸ“Œ Task
Status

Active

Version

2.0

Component

Code

Created by

πŸ‡¬πŸ‡§United Kingdom alexpott πŸ‡ͺπŸ‡ΊπŸŒ

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @alexpott
  • Merge request !107Remove the queue β†’ (Merged) created by alexpott
  • πŸ‡¬πŸ‡§United Kingdom alexpott πŸ‡ͺπŸ‡ΊπŸŒ

    I think we need an update function to delete any existing queue items and if there are queue items tell people to run the batch.

  • Pipeline finished with Success
    3 months ago
    Total: 234s
    #410139
  • Pipeline finished with Success
    3 months ago
    Total: 265s
    #411571
  • First commit to issue fork.
  • πŸ‡ͺπŸ‡ΈSpain marcoscano Barcelona, Spain

    Pushed a quick update function to clean up remaining items from the queue, if existing. Was that what you had in mind?

    Thanks!

  • πŸ‡¬πŸ‡§United Kingdom alexpott πŸ‡ͺπŸ‡ΊπŸŒ

    Yep exactly something like that... pushed a change to fix the test and to test the update.

  • Pipeline finished with Failed
    3 months ago
    Total: 284s
    #411834
  • Pipeline finished with Success
    3 months ago
    Total: 231s
    #411836
  • Pipeline finished with Skipped
    3 months ago
    #412087
  • πŸ‡ͺπŸ‡ΈSpain marcoscano Barcelona, Spain

    Great, let's go with this. Thank you! πŸ‘

  • Automatically closed - issue fixed for 2 weeks with no activity.

  • Status changed to Fixed about 1 month ago
  • πŸ‡³πŸ‡±Netherlands seanB Netherlands

    @alexpott @marcoscano A lot of great work to improve entity usage. Nice!
    This particular change seems to cause some issues for my current project though. If the batch somehow gets stuck or crashes, you have to start all over again. Yesterday it happened 3 times, after about 1,5 hours of running (we have a lot of plugins).

    Do you have some more background on the choice to remove the queue options? Is it for performance/maintenance only? Or is there a chance of the outcome just being wrong or inconsistent?

    Before we start creating our own drush command or write a patch it would be good to have some more background info.

  • πŸ‡ͺπŸ‡ΈSpain marcoscano Barcelona, Spain

    Hi @seanb! I am sorry this is causing issues for you.

    Maybe @alexpott can chime in with more context if needed, but my recollection is that:
    - The queue batch manager was a part of the codebase that had been added in the past as a stopgap solution to the poor performance when regenerating all info. It had been added without test coverage (sorry my bad), and had significant duplication with the normal batch manager
    - In the past few months Alex worked on a lot of great improvements to the performance of the module in general, and in particular to the batch regeneration process

    With the recent performance improvements (and knowing they are incompatible with how the queue works), it felt unnecessary to keep maintaining both batch managers, especially since they aren't particularly small/trivial. We felt the performance gains of the bulk insert batch manager were enough to justify having the Drush command always work as a standard batch operation, without the need to offer a queue mode as well.

Production build 0.71.5 2024