Out of memory because of too large library definitions

Created on 7 January 2025, 7 months ago

We have a large drupal setup, containing quite a few modules (contrib and custom), running 100+ sites.
PHP is set up with a memory limit of 256Mb.
A few weeks ago some sites started running out of memory on a cold cache.

After some investigation, the culprit was the "cache_discovery" of the "library_info:our_custom_theme".
It seems the library definition calculation has become so large for some of our sites that it needs 400Mb to calculate.
We have temporarily set our PHP memory_limit to 512Mb to mitigate the issue.

We did more investigation by trying to figure out which module was causing the most memory build-up and it turned out webform was responsible for half of the peak memory usage.

In Drupal\Core\Cache\CacheCollectorInterface\LibraryDiscovery::getLibrariesByExtension() we added a piece of code to test this:

  public function getLibrariesByExtension($extension) {
    if ($extension === 'webform') {
      return [];
    }

    ... original code ...

  }

Before this test we had a memory peak usage (cold cache) of 395Mb.
After the test only 195Mb.

I understand that 150Mb probably comes from our setup and we could look into that. But most likely we will end up concluding that there are no easy wins here.

Is this 200Mb of memory usage something that is expected to build the library cache for webform?
If so, how can we "fix" this so peak memory usage is way less?

πŸ› Bug report
Status

Needs work

Version

6.3

Component

Code

Created by

πŸ‡§πŸ‡ͺBelgium weseze

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @weseze
  • πŸ‡§πŸ‡ͺBelgium weseze

    I did some more digging and found the issue is the "webform_library_info_build()".
    This function loads all webform, checks them for custom CSS/JS and adds those as a library if needed.

    The website where we noticed the issue has 1000+ webforms. So trying to load them all leads to 200Mb of memory usage.

    I tried changing the code to load them 1-by-1 and clear the static entity caching between each, but this didn't help. The memory usage stayed the same. I thought this would have been a quick and easy "fix"... :(

    The better solution would be to split the code in 2 pieces:
    1/ general CSS/JSS assets need to be added once in a "shared" library for all webforms
    2/ only webforms that have specific CSS/JSS assets need to be loaded: this could still be an issue if there are many: not sure how to address that...

  • First commit to issue fork.
  • πŸ‡¬πŸ‡§United Kingdom catch

    The issue here is loading every webform and then looping over them to check if they have css or javascript.
    Config entities support entity queries, so it's possible to pre-filter with an entity query. For me this reduced the time taken for this hook from five seconds to about 230ms

  • πŸ‡¬πŸ‡§United Kingdom catch

    Adding a patch for a 6.2.x backport.

  • πŸ‡¬πŸ‡§United Kingdom catch
  • Pipeline finished with Failed
    11 days ago
    Total: 632s
    #551477
  • heddn Nicaragua

    Posted some feedback.

  • πŸ‡¬πŸ‡§United Kingdom catch

    Double checked and ::exists() definitely isn't an option here - it results in loading all the webforms in, so I think the <> '' logic is correct here.

  • heddn Nicaragua

    I'm good with this. LGTM

  • πŸ‡ΊπŸ‡ΈUnited States jrockowitz Brooklyn, NY

    The problem and solution makes a lot of sense to me. I am good with it

  • πŸ‡¨πŸ‡¦Canada Liam Morland Ontario, CA πŸ‡¨πŸ‡¦

    There is an unresolved thread; @catch, @heddn

  • πŸ‡¬πŸ‡§United Kingdom catch

    Went ahead and resolved the thread per #9.

  • Pipeline finished with Failed
    1 day ago
    Total: 650s
    #558835
Production build 0.71.5 2024