Manually clear cache keys from plugin managers with finite variations instead of using cache tags

Created on 23 January 2023, almost 2 years ago
Updated 16 January 2024, 11 months ago

Problem/Motivation

Follow-up from πŸ› ChainedFastBackend invalidates all items when cache tags are invalidated Fixed .

Several core plugin managers use cache tags for invalidation - for example the element info cache depends on the theme, so an 'element_info' cache tag is added, and that way the cache can be invalidated for all themes using the tag.

This is also done in at least some places by language.

This adds some overhead to cache gets, because the cache tag needs to be fetched. However, we know that the cache tag is not cleared arbitrarily, it's only used in ::clearCachedDefinitions()

element_info_build
* theme_registry
* library_info

Steps to reproduce

Proposed resolution

Instead of using a cache tag, loop over the list of themes in ::clearCachedDefinitions(), clearing each cid individually.

Remaining tasks

Anything straightforward I think we could do in one issue. There might be some other discovery mechanisms (breakpoints etc.) where we want to open spin-off issues.

User interface changes

API changes

Data model changes

Release notes snippet

πŸ“Œ Task
Status

Fixed

Version

11.0 πŸ”₯

Component
BaseΒ  β†’

Last updated about 2 hours ago

Created by

πŸ‡¬πŸ‡§United Kingdom catch

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • Issue created by @catch
  • πŸ‡¨πŸ‡­Switzerland berdir Switzerland

    there are some cache tag invalidations for library_info and theme_registry, for example \Drupal\system\EventSubscriber\ConfigCacheTag::onSave. That's why I was asking whether we considering that an API and therefore BC break.

  • πŸ‡«πŸ‡·France andypost

    How it will clear theme on its uninstall?

  • πŸ‡¬πŸ‡§United Kingdom catch

    there are some cache tag invalidations for library_info and theme_registry, for example \Drupal\system\EventSubscriber\ConfigCacheTag::onSave. That's why I was asking whether we considering that an API and therefore BC break.

    hmm I think that's actually wrong, and that it should be doing that via ::clearCachedDefinitions() on the plugin manager, if that's what it wants to clear. However we might want to open a spin-off issue for that to make that change first.

  • πŸ‡¨πŸ‡­Switzerland berdir Switzerland

    > How it will clear theme on its uninstall?

    It won't be requested anymore when it's uninstalled so that doesn't really matter. That said, module uninstall empties all cache bins and we could do the same for themes if we'd want to.

  • Status changed to Needs review almost 2 years ago
  • πŸ‡¨πŸ‡­Switzerland berdir Switzerland

    Some work, we'll see how well it works. Didn't touch the language plugin managers yet. library_info usage is too dynamic, we can't replace that one.

  • πŸ‡«πŸ‡·France andypost

    fix to see test result

  • πŸ‡«πŸ‡·France andypost

    23 failed tests

  • Status changed to Needs work almost 2 years ago
  • πŸ‡¨πŸ‡­Switzerland berdir Switzerland
  • Status changed to Needs review almost 2 years ago
  • πŸ‡¨πŸ‡­Switzerland berdir Switzerland

    I missed the module theme build cache.

  • Status changed to Needs work almost 2 years ago
  • The Needs Review Queue Bot β†’ tested this issue. It fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

    Apart from a re-roll or rebase, this issue may need more work to address feedback in the issue or MR comments. To progress an issue, incorporate this feedback as part of the process of updating the issue. This helps other contributors to know what is outstanding.

    Consult the Drupal Contributor Guide β†’ to find step-by-step guides for working with issues.

  • Status changed to Needs review almost 2 years ago
  • πŸ‡«πŸ‡·France andypost

    Fixed CS

  • Status changed to Needs work almost 2 years ago
  • πŸ‡¨πŸ‡­Switzerland berdir Switzerland

    Thanks for fixing my cs issues ;)

    Down to two failures now, great, and that seems to be just a unit test that we'll need to update.

    And the per-language plugin caches need to be updated too.

  • Status changed to Needs review almost 2 years ago
  • πŸ‡¨πŸ‡­Switzerland berdir Switzerland

    Fixed the unit test and updated some other plugin managers that use cache tags.

    * help topics has an extension cache tag, but all plugins are invalidated when modules are installed or uninstalled, so that shouldn't be needed at all.
    * contextual links, removed for now, but this also has group caches that do _not_ use the tag, so I think those are already broken. The thing is that I doubt that these are needed at all. it also doesn't do a reset of the data on the property. The only calculation is a single loop and checking them against the group, that should be quite fast? And I'm pretty sure we load all when a definition is then fetched for them.
    * local actions

    Kept it for local task manager, as that has per-route caches.

  • @berdir opened merge request.
  • πŸ‡«πŸ‡·France andypost

    Help topics having twig extensions for routes/links and search module integration required hack for reindexing

  • πŸ‡¨πŸ‡­Switzerland berdir Switzerland

    All plugin types depend on which modules are enabled, that's why w clear them all through the plugin.cache_clearer service. However, we're not yet doing that for themes, which the cache tag supported. I added the cache clear call for that case instead, this doesn't cover uninstall yet, we probably need that too.

  • First commit to issue fork.
  • πŸ‡¦πŸ‡ΊAustralia VladimirAus Brisbane, Australia
  • Status changed to Needs work almost 2 years ago
  • πŸ‡ΊπŸ‡ΈUnited States smustgrave

    Wonder if a simple change record could be created to announce the new function clearCachedDefinitions

  • Status changed to Needs review almost 2 years ago
  • πŸ‡¨πŸ‡­Switzerland berdir Switzerland

    > clearCachedDefinitions

    This is not a new function, it's just implementing an existing API of plugin managers.

    But yes, this does need a change record, to announce the cache tags that are no longer available for invalidation and that instead those (existing) methods must be used. This should be very rare, there's a single case in core.

    Somehow the previous changes from the patches didn't make it into the merge request, added that back now. Setting back to needs review to run the tests, this also needs reviews.

  • Status changed to Needs work almost 2 years ago
  • πŸ‡ΊπŸ‡ΈUnited States smustgrave

    Seems to have a CI failure in the MR.

    Ignored error pattern #cache tag might be unclear and does not contain
    the cache key in it.# was not matched in reported errors.
    -- --------------------------------------------------------------

  • Status changed to Needs review almost 2 years ago
  • πŸ‡¨πŸ‡­Switzerland berdir Switzerland

    Fixed that I think.

  • Status changed to RTBC almost 2 years ago
  • πŸ‡¦πŸ‡ΊAustralia VladimirAus Brisbane, Australia

    +1. Happy with the changes and tests.

  • πŸ‡¨πŸ‡­Switzerland berdir Switzerland

    Still a bit of work left, and we need to create the change record.

  • last update over 1 year ago
    Custom Commands Failed
  • last update over 1 year ago
    Custom Commands Failed
  • Status changed to Needs review over 1 year ago
  • πŸ‡¨πŸ‡­Switzerland berdir Switzerland

    Rerolled and updated constructors. Created https://www.drupal.org/node/3355227 β†’ .

  • πŸ‡«πŸ‡·France andypost
      Line   core/lib/Drupal/Core/Theme/Registry.php                              
     ------ --------------------------------------------------------------------- 
      770    Call to static method invalidateTags() on an unknown class           
             Drupal\Core\Theme\Cache.                                             
             πŸ’‘ Learn more at https://phpstan.org/user-guide/discovering-symbols  
  • last update over 1 year ago
    Custom Commands Failed
  • πŸ‡¨πŸ‡­Switzerland berdir Switzerland

    Fixed that in the gitlab editor. FWIW, I'm not convinced that cache tag invalidation is now in a better place. I think both module and theme install and uninstall should consistently clear caches, but that's not in scope of this issue.

  • Status changed to Needs work over 1 year ago
  • πŸ‡«πŸ‡·France andypost

    There's still CS errors and it needs follow-up for #29

  • Status changed to Needs review over 1 year ago
  • πŸ‡¨πŸ‡­Switzerland berdir Switzerland

    Meh. ;)

  • last update over 1 year ago
    29,283 pass
  • πŸ‡«πŸ‡·France andypost

    Left few review comments, it looks mostly ready

  • last update over 1 year ago
    Custom Commands Failed
  • last update over 1 year ago
    29,300 pass
  • πŸ‡¨πŸ‡­Switzerland berdir Switzerland

    Thank for the review, cleaned up ElementInfoManager and also added the trait for the removed property.

  • last update over 1 year ago
    29,366 pass
  • Status changed to Needs work over 1 year ago
  • πŸ‡ΊπŸ‡ΈUnited States smustgrave

    Seems there are 2 open threads on the MR still.

    Think this should be updated for 10.2 now also?

  • Status changed to Needs review over 1 year ago
  • πŸ‡¨πŸ‡­Switzerland berdir Switzerland

    As commented on the two open and related threads, I haven't seen mixed constructor declarations in core before, are you sure that is a thing we do? Setting to needs review to get more feedback.

  • πŸ‡«πŸ‡·France andypost

    There's in core https://git.drupalcode.org/project/drupal/-/blob/10.1.x/core/lib/Drupal/...

    Moreover there's more

    $ git grep ' __construct('|grep -E '( \w*\|\w* )'
    core/lib/Drupal/Component/DependencyInjection/ReverseContainer.php:35:  public function __construct(private readonly Container|SymfonyContainer $serviceContainer) {
    core/modules/block/src/Plugin/migrate/process/BlockTheme.php:46:  public function __construct(array $configuration, $plugin_id, $plugin_definition, Config|MigrationInterface $theme_config, array|Config $themes) {
    core/modules/node/src/Form/NodeRevisionDeleteForm.php:76:  public function __construct(EntityStorageInterface $node_storage, EntityStorageInterface $node_type_storage, AccessManagerInterface|Connection $access_manager, DateFormatterInterface $date_formatter) {
    

    So I added commit with this change

  • last update over 1 year ago
    Custom Commands Failed
  • last update over 1 year ago
    19,911 pass, 2,869 fail
  • last update over 1 year ago
    19,911 pass, 2,869 fail
  • last update over 1 year ago
    29,380 pass
  • πŸ‡«πŸ‡·France andypost

    @Berdir thanks for review, fixed wrong condition

  • Status changed to Needs work over 1 year ago
  • πŸ‡ΊπŸ‡ΈUnited States smustgrave

    All the threads seem to be resolved.

    I was looking at the change record and think it would help to mention that ConfigCacheTag now requires \Drupal\Core\Theme\Registry and ElementInfoManager no longer takes \Drupal\Core\Cache\CacheTagsInvalidatorInterface

  • last update over 1 year ago
    29,802 pass
  • Status changed to Needs review over 1 year ago
  • πŸ‡¨πŸ‡­Switzerland berdir Switzerland

    Rebased the MR on 11.x. πŸ› ChainedFastBackend invalidates all items when cache tags are invalidated Fixed is in, so would be nice to land this as well to fully benefit from it.

    #38: Personally, I find change records for trivial constructor changes for classes that are rarely if ever subclassed not useful. It doesn't belong in the change record that's about the actual change as it's related to that, and a separate one just for that is IMHO just noise and makes it harder to find relevant changes. The only afffected contrib module seems to be bootstrap for Registry, there are some subclasses of ElementInfoManager but they don't change the constructor.

    But if that's blocking getting this to RTBC I'll create one.

  • πŸ‡¬πŸ‡§United Kingdom catch

    Agreed we don't need change records for constructor changes. Not really here but will try to review this again soon.

  • πŸ‡¨πŸ‡­Switzerland berdir Switzerland

    Did a little bit of profiling, pretty much as expected. Not a big change, but with render caching disabled on standard install on frontpage as admin, we're saving 3 calls to DatabaseCacheTagsChecksum::getTagInvalidationCounts(), each of them did a lookup for a unique and otherwise not used cache tag, so we save 3 queries. There are more plugin managers being changed, but they're not all being used in this cenario (link relations, help?, migration, ..).

  • Status changed to RTBC over 1 year ago
  • πŸ‡ΊπŸ‡ΈUnited States smustgrave

    Putting in committer queue.
    Also removing CR updates tag per #41

  • last update over 1 year ago
    29,807 pass
    • catch β†’ committed 82d3e957 on 11.x
      Issue #3335768 by Berdir, andypost, smustgrave, catch: Manually clear...
  • Status changed to Fixed over 1 year ago
  • πŸ‡¬πŸ‡§United Kingdom catch

    Updated the deprecation messages to 10.2 on commit. Committed/pushed to 10.2.x, thanks!

  • Automatically closed - issue fixed for 2 weeks with no activity.

  • Status changed to Fixed 11 months ago
  • πŸ‡§πŸ‡ͺBelgium wim leers Ghent πŸ‡§πŸ‡ͺπŸ‡ͺπŸ‡Ί

    FYI this caused a regression in migrate_plus with no deprecation error: #3413533-5: Fix failing tests on HEAD β†’ .

Production build 0.71.5 2024