Add the ability to bulk insert to EntityUsage service

Created on 14 January 2025, about 1 month ago

Problem/Motivation

On a large site creating the initial entity usage table is slow. We can massively improve this by bulk inserting information into the entity_usage table and not doing unnecessary deletes.

Proposed resolution

Add the ability to do bulk inserts to \Drupal\entity_usage\EntityUsage and leverage it from \Drupal\entity_usage\EntityUsageBatchManager::updateSourcesBatchWorker() when the user has chosen to not keep existing records or the table is empty for the entity type.

Remaining tasks

User interface changes

None

API changes

TBD

Data model changes

None

πŸ“Œ Task
Status

Active

Version

2.0

Component

Code

Created by

πŸ‡¬πŸ‡§United Kingdom alexpott πŸ‡ͺπŸ‡ΊπŸŒ

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @alexpott
  • Pipeline finished with Failed
    about 1 month ago
    Total: 193s
    #395610
  • Pipeline finished with Failed
    about 1 month ago
    Total: 229s
    #395619
  • Pipeline finished with Failed
    about 1 month ago
    Total: 233s
    #395628
  • Pipeline finished with Failed
    about 1 month ago
    Total: 224s
    #395631
  • Pipeline finished with Failed
    about 1 month ago
    Total: 220s
    #395850
  • Pipeline finished with Failed
    about 1 month ago
    Total: 208s
    #395854
  • πŸ‡¬πŸ‡§United Kingdom alexpott πŸ‡ͺπŸ‡ΊπŸŒ

    I've tested this on a project with thousand of nodes and paragraphs and many many revisions - some nodes with over 100! And it completes the bulk load of the entity_usage table much much quicker. Especially when combined with https://www.drupal.org/project/entity_lru_cache β†’ which fixes out-of-control memory usage due to loading entities. There is still much that can be improved in follow-ups. For example:

    • process entities in parallel using child processes
    • Not recalculate the usage data if the previous revision and current revision have not changed (this could benefit runtime entity saving too if we were smart)

    I need to confirm we have test coverage of all the code before we proceed in actually merging this.

  • Pipeline finished with Success
    about 1 month ago
    Total: 234s
    #396515
  • Pipeline finished with Success
    about 1 month ago
    Total: 305s
    #396786
  • Pipeline finished with Failed
    about 1 month ago
    Total: 215s
    #396869
  • Pipeline finished with Failed
    about 1 month ago
    Total: 263s
    #396967
  • Pipeline finished with Success
    about 1 month ago
    #396970
  • Pipeline finished with Skipped
    about 1 month ago
    #403481
  • First commit to issue fork.
  • πŸ‡ͺπŸ‡ΈSpain marcoscano Barcelona, Spain

    Nice work! Thanks for helping!

    I have opened πŸ“Œ Allow users to clear the bulk inserting state flag on the UI Active as a follow-up so we can add an option to clear the state flag on the UI, but that isn't a huge deal IMO.

  • Automatically closed - issue fixed for 2 weeks with no activity.

Production build 0.71.5 2024