Out of memory because of too large library definitions

Created on 7 January 2025, 3 months ago

We have a large drupal setup, containing quite a few modules (contrib and custom), running 100+ sites.
PHP is set up with a memory limit of 256Mb.
A few weeks ago some sites started running out of memory on a cold cache.

After some investigation, the culprit was the "cache_discovery" of the "library_info:our_custom_theme".
It seems the library definition calculation has become so large for some of our sites that it needs 400Mb to calculate.
We have temporarily set our PHP memory_limit to 512Mb to mitigate the issue.

We did more investigation by trying to figure out which module was causing the most memory build-up and it turned out webform was responsible for half of the peak memory usage.

In Drupal\Core\Cache\CacheCollectorInterface\LibraryDiscovery::getLibrariesByExtension() we added a piece of code to test this:

  public function getLibrariesByExtension($extension) {
    if ($extension === 'webform') {
      return [];
    }

    ... original code ...

  }

Before this test we had a memory peak usage (cold cache) of 395Mb.
After the test only 195Mb.

I understand that 150Mb probably comes from our setup and we could look into that. But most likely we will end up concluding that there are no easy wins here.

Is this 200Mb of memory usage something that is expected to build the library cache for webform?
If so, how can we "fix" this so peak memory usage is way less?

🐛 Bug report

Status

Needs work

Version

6.3

Component

Code

Created by

🇧🇪Belgium weseze

Live updates comments and jobs are added and updated live.

Comments & Activities

Issue created by @weseze
Comment 3 months ago →
🇧🇪Belgium weseze
I did some more digging and found the issue is the "webform_library_info_build()".
This function loads all webform, checks them for custom CSS/JS and adds those as a library if needed.

The website where we noticed the issue has 1000+ webforms. So trying to load them all leads to 200Mb of memory usage.

I tried changing the code to load them 1-by-1 and clear the static entity caching between each, but this didn't help. The memory usage stayed the same. I thought this would have been a quick and easy "fix"... :(

The better solution would be to split the code in 2 pieces:
1/ general CSS/JSS assets need to be added once in a "shared" library for all webforms
2/ only webforms that have specific CSS/JSS assets need to be loaded: this could still be an issue if there are many: not sure how to address that...

contrib.social Blog FAQ Discussions

Production build 0.71.5 2024