Generate and cache library info on hook_rebuild

Created on 7 February 2024, 11 months ago
Updated 23 April 2024, 8 months ago

Problem/Motivation

If certain conditions coincide, such as high load and the absence of certain keys in the cache, the site may break. The CSP module checks the cache on every request and tries to add a key if one is missing. If the cache_default table is in use, the query waits until the lock is released before it can be executed again. Under high load, the cache does not have time to be created and a new request to add an entry to the database is added to the waiting queue.

NOTICE: PHP message: Uncaught PHP Exception Drupal\Core\Database\DatabaseExceptionWrapper: "SQLSTATE[HY000]: General error: 1205 Lock wait timeout exceeded; try restarting transaction: INSERT INTO "cache_default" ("cid", "expire", "created", "tags", "checksum", "data", "serialized") VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6) ON DUPLICATE KEY UPDATE "cid" = VALUES("cid"), "expire" = VALUES("expire"), "created" = VALUES("created"), "tags" = VALUES("tags"), "checksum" = VALUES("checksum"), "data" = VALUES("data"), "serialized" = VALUES("serialized"); Array
(
    [:db_insert_placeholder_0] => csp:libraries:core
    [:db_insert_placeholder_1] => -1
    [:db_insert_placeholder_2] => 1706876727.955
    [:db_insert_placeholder_3] => library_info
    [:db_insert_placeholder_4] => 1022
    [:db_insert_placeholder_5] => a:61:{s:8:"backbone";a:0:{}s:17:"internal.backbone";a:0:{}s:8:"ckeditor";a:0:{}s:9:"ckeditor5";a:0:{}s:23:"ckeditor5.editorClassic";a:0:{}s:25:"ckeditor5.editorDecoupled";a:0:{}s:20:"ckeditor5.essentials";a:0:{}s:17:"ckeditor5.heading";a:0:{}s:15:"ckeditor5.basic";a:0:{}s:27:"ckeditor5.specialCharacters";a:0:{}s:20:"ckeditor5.blockquote";a:0:{}s:15:"ckeditor5.image";a:0:{}s:14:"ckeditor5.link";a:0:{}s:14:"ckeditor5.list";a:0:{}s:24:"ckeditor5.horizontalLine";a:0:{}s:21:"ckeditor5.htmlSupport";a:0:{}s:19:"ckeditor5.alignment";a:0:{}s:22:"ckeditor5.removeFormat";a:0:{}s:25:"ckeditor5.pasteFromOffice";a:0:{}s:16:"ckeditor5.indent";a:0:{}s:23:"ckeditor5.sourceEditing";a:0:{}s:15:"ckeditor5.table";a:0:{}s:18:"ckeditor5.language";a:0:{}s:19:"ckeditor5.codeBlock";a:0:{}s:15:"ckeditor5.style";a:0:{}s:22:"ckeditor5.translations";a:0:{}s:10:"css.escape";a:0:{}s:6:"drupal";a:0:{}s:14:"drupalSettings";a:0:{}s:18:"drupal.active-link";a:0:{}s:11:"drupal.ajax";a:0:{}s:15:"drupal.announce";a:0:{}s:6:"loadjs";a:0:{}s:19:"drupal.autocomplete";a:0:{}s:17:"drupal.array.find";a:0:{}s:21:"drupal.array.includes";a:0:{}s:12:"drupal.batch";a:0:{}s:15:"drupal.checkbox";a:0:{}s:15:"drupal.collapse";a:0:{}s:18:"drupal.customevent";a:0:{}s:11:"drupal.date";a:0:{}s:15:"drupal.debounce";a:0:{}s:13:"drupal.dialog";a:0:{}s:18:"drupal.dialog.ajax";a:0:{}s:15:"drupal.displace";a:0:{}s:17:"drupal.dropbutton";a:0:{}s:22:"drupal.element.closest";a:0:{}s:22:"drupal.element.matches";a:0:{}s:18:"drupal.entity-form";a:0:{}s:11:"drupal.form";a:0:{}s:19:"drupal.machine-name";a:0:{}s:14:"drupal.message";a:0:{}s:23:"drupal.nodelist.foreach";a:0:{}s:20:"drupal.object.assign";a:0:{}s:15:"drupal.progress";a:0:{}s:13:"drupal.states";a:0:{}s:22:"drupal.string.includes";a:0:{}s:21:"drupal.tabbingmanager";a:0:{}s:16:"drupal.tabledrag";a:0:{}s:18:"drupal.tableheader";a:0:{}s:22:"drupal.tableresponsive";a:0:{}}
    [:db_insert_placeholder_6] => 1
)
" at /var/www/html/web/core/modules/mysql/src/Driver/Database/mysql/ExceptionHandler.php line 53

Steps to reproduce

  1. Enable and configure CSP module
  2. Generate generate load on the site (10 rpc is enough)
  3. Regenerate cache (drush cr)

Proposed resolution

Implement hook_rebuild() to warm empty cache values for library info.

Remaining tasks

User interface changes

API changes

Data model changes

Feature request
Status

Fixed

Version

2.0

Component

Code

Created by

🇷🇺Russia aa2007

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @aa2007
  • Merge request !31[issue 3419681] move caching to the hook_rebuild → (Closed) created by aa2007
  • Pipeline finished with Success
    11 months ago
    Total: 584s
    #91091
  • Pipeline finished with Canceled
    11 months ago
    Total: 124s
    #91100
  • Pipeline finished with Success
    11 months ago
    Total: 149s
    #91102
  • Pipeline finished with Success
    11 months ago
    Total: 154s
    #91111
  • Pipeline finished with Success
    11 months ago
    Total: 153s
    #91119
  • Pipeline finished with Success
    11 months ago
    Total: 149s
    #91145
  • Pipeline finished with Success
    11 months ago
    Total: 150s
    #91183
  • Status changed to Needs work 11 months ago
  • 🇨🇦Canada gapple

    Leveraging hook_rebuild to warm the cache is a good suggestion, but the responsibility of storing the data to cache should stay in the service, since there are occasions other than a full cache rebuild where it's possible that the items are not in the cache. If they aren't recreated on demand if missing, then every future page request is going to parse the site's library info files.

    If a site is having issues with MySQL locks on cache tables, it seems like there's many places where that could occur, so my thoughts would be:
    - The database server's isolation level might need to be adjusted ( docs )
    - The site might benefit from moving to a different cache backend like memcache or redis

    ----

    I'm not sure if the database cache backend can be configured to ignore a timeout when writing a value, since a page can render fine after skipping storage (and hopefully it's taken care of by the request that's holding the lock).

  • Pipeline finished with Success
    11 months ago
    Total: 182s
    #91200
  • Pipeline finished with Success
    11 months ago
    Total: 186s
    #91228
    • gapple committed 3d230075 on 2.x
      Issue #3419681: Parse and cache library sources on hook_rebuild
      
    • gapple committed 5deccca9 on 8.x-1.x
      Issue #3419681: Parse and cache library sources on hook_rebuild
      
  • Status changed to Fixed 11 months ago
  • Automatically closed - issue fixed for 2 weeks with no activity.

  • 🇷🇴Romania zoltanb

    Hi

    FYI with the patch applied on 1.30 release, i get

    [warning] foreach() argument must be of type array|object, null given DefinitionDiscoveryFactory.php:73
     [success] Cache rebuild complete.

    after rebuilding the cache.

Production build 0.71.5 2024