Remove the aggregate stale file threshold and state entry

Created on 3 August 2022, almost 2 years ago
Updated 21 December 2023, 6 months ago

Problem/Motivation

CSS and JavaScript aggregation previously generated files inline during HTML page requests. If the file was requested without a PHP request to the page, and didn't exist, there was no way to create it and you'd get unstyled pages or broken JavaScript. This could happen just from a clear of js/css files on the server and then HTML cached in a CDN.

However it also had to periodically delete css and js files to avoid a potentially infinite collection of files.

To get around this only files older than 30 days are deleted when there's a cache clear.

However, following 🐛 Stampedes and cold cache performance issues with css/js aggregation Fixed , as long as a file is valid for libraries available on the site, we can recreate it independent of the 'parent' HTML request, and don't have to worry about the race condition.

Additionally, we store all the created files in state, this is necessary because the asset filename is based on the content of the files themselves, and that would be too expensive to recalculate. The asset filename with the new system is cheaper to recreate, so we don't really need this extra level of i/o caching there at all.

Steps to reproduce

Proposed resolution

Remove the css.cache_files and js.cache_files state entries.

Remove the system.performance stale file threshold.

Remaining tasks

User interface changes

API changes

Data model changes

Release notes snippet

📌 Task
Status

Fixed

Version

10.1

Component
Asset library 

Last updated about 18 hours ago

No maintainer
Created by

🇬🇧United Kingdom catch

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • 🇬🇧United Kingdom catch

    Random failure.

  • Status changed to Needs work over 1 year ago
  • The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

    Apart from a re-roll or rebase, this issue may need more work to address feedback in the issue or MR comments. To progress an issue, incorporate this feedback as part of the process of updating the issue. This helps other contributors to know what is outstanding.

    Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

  • Status changed to Needs review over 1 year ago
  • 🇮🇳India ameymudras

    Rerolled the patch

  • Status changed to Needs work over 1 year ago
  • 🇺🇸United States smustgrave

    Running PHPStan on *all* files.
    ------ ---------------------------------------------------------------------------
    Line core/tests/Drupal/Tests/Core/Asset/CssCollectionOptimizerLazyUnitTest.php
    ------ ---------------------------------------------------------------------------
    55 Class Drupal\Core\Asset\CssCollectionOptimizerLazy constructor
    invoked with 11 parameters, 10 required.
    115 Class Drupal\Core\Asset\CssCollectionOptimizerLazy constructor
    invoked with 11 parameters, 10 required.

  • Status changed to Needs review over 1 year ago
  • 🇮🇳India mrinalini9 New Delhi

    Updated patch #23 by addressing #24, please review it.

    Thanks!

  • Status changed to Needs work over 1 year ago
  • 🇬🇧United Kingdom catch

    The interdiff looks good.

    The last thing here is to deal with #17 and #18, I think we should probably remove the post update hook from this patch, and open a follow-up against Drupal 11 to add it there - can just have that part of the patch attached. Marking needs work for that.

  • Status changed to Needs review 12 months ago
  • last update 12 months ago
    Custom Commands Failed
  • 🇫🇮Finland lauriii Finland
  • 🇬🇧United Kingdom catch

    Re-rolled. Thinking about #17/#18 more I think if someone somewhere really is using the old classes, they should be responsible for cleaning up the state entries too, so leaving the update in. Crediting people from the other issue.

    A workaround if you wanted this in 10.1.0 would be to set stale_file_threshold to 0.

    Bumping to major since this is a DX issue for local development, shame it stalled.

  • last update 12 months ago
    29,801 pass
  • 🇬🇧United Kingdom catch

    Fixing cs issues.

    We could also backport the ::deleteAll() changes to 10.0 without the deprecations. That would improve things for local development and deployments.

  • Status changed to RTBC 12 months ago
  • 🇫🇮Finland lauriii Finland

    To me #32 sounds reasonable. Even if @alexpott disagrees, it seems fine to run the post update now and we could have another post update later to clean it up again if he feels strongly about this. I don't think there are any usages in contrib.

    +++ b/core/lib/Drupal/Core/Asset/CssCollectionOptimizerLazy.php
    @@ -135,27 +131,15 @@ public function optimize(array $css_assets, array $libraries) {
       public function getAll() {
    

    Went through contrib uses of this class and didn't see any calls for this. Seems fine to make this change and deprecate the method.

    +1 for backporting the changes to \Drupal\Core\Asset\CssCollectionOptimizerLazy::deleteAll to 10.1.x.

  • Open in Jenkins → Open on Drupal.org →
    Environment: PHP 8.1 & MySQL 5.7
    last update 12 months ago
    29,443 pass
  • 🇬🇧United Kingdom catch

    Here's the 10.1 backport.

  • 🇫🇷France andypost
    +++ b/core/lib/Drupal/Core/Asset/AssetCollectionOptimizerInterface.php
    @@ -25,6 +25,11 @@ public function optimize(array $assets, array $libraries);
    +   * @deprecated in drupal:10.2.0 and is removed from drupal:11.1.0. There is
    
    +++ b/core/lib/Drupal/Core/Asset/CssCollectionOptimizerLazy.php
    @@ -135,27 +131,15 @@ public function optimize(array $css_assets, array $libraries) {
    +    @trigger_error(__METHOD__ . ' is deprecated in drupal:10.2.0 and will be removed in drupal:11.0.0, there is no replacement. See https:// www.drupal.org/node/3301744', E_USER_DEPRECATED);
    
    +++ b/core/lib/Drupal/Core/Asset/JsCollectionOptimizerLazy.php
    @@ -150,26 +146,15 @@ public function optimize(array $js_assets, array $libraries) {
    +    @trigger_error(__METHOD__ . ' is deprecated in drupal:10.2.0 and will be removed in drupal:11.0.0, there is no replacement. See https:// www.drupal.org/node/3301744', E_USER_DEPRECATED);
    

    Could be fixed on commit - s/11.1/11.0

  • 🇺🇸United States mherchel Gainesville, FL, US

    Will this fix the issue in 10.1 where if a user clears cache, it doesn't refresh the aggregate CSS/JS that's pulled down?

  • 🇺🇸United States nsciacca Connecticut

    @mherchel - I just applied this to my 10.1 and other than a warning "] rmdir(/files/css): Directory not empty FileSystem.php:267" it did appear to fix the aggregate CSS/JS on my Pantheon deploy.

  • 🇬🇧United Kingdom catch

    @mherchel yes it should.

  • 🇺🇸United States mherchel Gainesville, FL, US

    @mherchel yes it should.

    Just tested this out, and it doesn't appear to be working.

    Not sure if this blocks this issue, or if this should be reported in a followup.

    Testing steps

    1. Ensure aggregated CSS/JS, caching enabled etc.
    2. View source. Copy filename including querystring and save it (It looks something like

      /sites/default/files/css/css_hegpbZvWLCegz1eocO6sYJ0X_AGboa15OU4bOdIZPaI.css?delta=0&language=en&theme=olivero&include=eJx9UFuSgyAQvBDqGfYkqQF6lV1kqBmI6-0XEzWVVCo_MPQDmnYsGLzUTLHPwqNA1bgN5ORwnyKT_9nRXephud4FqeCvVIoH9UC6GNKvGl21YB4sKZ7uIFfCFTeR4dhG4SHzAoHv7NrZyO5BfAPeXAMWHW5rP7OvESdfaNTzkOgaRiqBU6doeTzJepIKEjd1S_B4Z8gS5jfyRCK8PHcQNEdqHRTmaEmGfTdVIYdom_vQMuz9hVaOpEbohDxBfHPXzVql07K2MsYH8vRcIWsbO1OiEfK5-COJTizF1XJIjrMhP4d0eQneFwE-UP3ErZLX7_ZQRxlfm83YMF5yyBiO4R-XNOx7

    3. Clear cache
    4. View source again and compare the aggregate filename/querystring against what I checked before.
    5. I expect the filename or querystring to change but it doesn't. I also manually modified Olivero's CSS (to intentionally make it out of date), but that had no affect.
  • 🇬🇧United Kingdom catch

    @mherchel can you check if drush cr deletes the aggregate file from disk? And can you try a hard browser refresh to see if you get the new contents from it once it's recreated?

  • 🇺🇸United States mherchel Gainesville, FL, US

    Yep. Verified its working 🙌

    Testing steps

    1. Verified path of CSS aggregate file
    2. Ran drush cr
    3. noted that entire CSS directory is now removed from sites/files
    4. modified a CSS file in Olivero
    5. Refreshed the path to the CSS file
    6. Noted that the new file is created, and it has the updated content.

    Thank you!

  • 🇫🇮Finland lauriii Finland
    +++ b/core/lib/Drupal/Core/Asset/CssCollectionOptimizerLazy.php
    @@ -142,20 +142,7 @@ public function getAll() {
    -    $this->state->delete('drupal_css_cache_files');
    

    Shouldn't we keep this?

  • 🇬🇧United Kingdom catch

    @lauriii it's no longer set by anything non-deprecated in core so it's redundant code in the new classes.

  • Open in Jenkins → Open on Drupal.org →
    Environment: PHP 8.1 & MySQL 5.7
    last update 12 months ago
    29,443 pass
  • last update 12 months ago
    29,797 pass
  • 🇬🇧United Kingdom catch

    Discussed #43/#44 with @lauriii - while the delete should be a no-op, in the interests of keeping changes to the absolute minimum in 10.1, adding it back.

  • last update 12 months ago
    29,439 pass
    • lauriii committed 405f8b33 on 11.x
      Issue #3301573 by catch, ravi.shankar, mrinalini9, mherchel, lauriii,...
    • lauriii committed f2933175 on 10.1.x
      Issue #3301573 by catch, ravi.shankar, mrinalini9, mherchel, lauriii,...
  • Status changed to Fixed 12 months ago
  • 🇫🇮Finland lauriii Finland

    Committed 405f8b33a7 and pushed to 11.0.x. Also committed the 10.1.x patch. Thanks!

  • Status changed to Active 12 months ago
  • 🇷🇺Russia BadWebCoder

    #45, It works when I clear the cache via the admin toolbar, but not when I do it via Drush.
    When I run drush cr I get the following messages:

    [notice] The file assets://css was not deleted because it does not exist.
    [notice] The file assets://js was not deleted because it does not exist.
    

    CSS and JS are not removed and new files are not created.

  • 🇷🇺Russia BadWebCoder

    #45, works when I run cache clear via admin toolbar, but doesn't work via Drush.
    When I run drush cr command I get the following messages:

    [notice] The file assets://css was not deleted because it does not exist.
    [notice] The file assets://js was not deleted because it does not exist.
    

    As a result, old CSS and JS files are not deleted, and new ones are not created.

  • Status changed to Fixed 12 months ago
  • 🇬🇧United Kingdom catch

    @BadWebCoder please open a new issue. The existing code was already deleting files in that directory, just not the directory itself, so if drush isn't able to find the directory now, it probably wasn't able to find it beforehand either.

  • 🇳🇬Nigeria chike Nigeria

    I did get the notices @BadWebCoder mentioned when I cleared cache with Drush,

    $ vendor/bin/drush cr
     [notice] The file assets://css was not deleted because it does not exist.
     [notice] The file assets://js was not deleted because it does not exist.

    But when I refreshed the page, my CSS changes did apply to the page.

    Off-topic: I can also see that some of the forum topics I added on the site which I saw on the PC (it is a fresh install), viewing the site on a phone I can't see the most recent forum topics. Content this time and not CSS or JS.

  • 🇳🇬Nigeria chike Nigeria

    Kindly ignore the off-topic above. It was my fault. I had wrong forum access permissions set.

  • 🇳🇬Nigeria chike Nigeria

    If CSS aggregation is turned on, CSS changes don't show up on the site and the site remains in the past never changing, whether caches are cleared in Drush or UI.

  • 🇷🇺Russia Chi

    The existing code was already deleting files in that directory, just not the directory itself

    @catch, seems it is deleting the directory as well.

    https://git.drupalcode.org/project/drupal/-/blob/10.1.1/core/lib/Drupal/...

  • 🇮🇹Italy FiNeX

    Currently it's also deleted the JS directory and both are never recreated (tested on both 10.1.1 and 10.1.x-dev).

  • Automatically closed - issue fixed for 2 weeks with no activity.

  • Status changed to Fixed 11 months ago
  • 🇺🇸United States edmund.dunn Olympia, WA

    We are having the same issue as @FiNeX in #56.

  • 🇳🇿New Zealand quietone New Zealand

    published the change record

  • 🇧🇪Belgium Wim Leers Ghent 🇧🇪🇪🇺

    AFAICT this incorrectly removed system.performance:stale_file_threshold from config schema. It should have deprecated that instead? 🤔 (That is possible since #2997100: Introduce a way to deprecate config schemas .)

Production build 0.69.0 2024