Drupal\Core\Config\FileStorage::getAllCollectionNamesHelper should use some caching

Created on 25 September 2023, about 1 year ago
Updated 3 March 2024, 8 months ago

Problem/Motivation

I have a project with a lot of configurations (3 037, mostly because of multiple languages). The site installation from existing configuration (drush site:install --existing-config) takes about 10 minutes. I did profile the installation and this is the TOP 10 slow calls:

As you can see, Drupal\Core\Config\FileStorage->getAllCollectionNamesHelper() is the slowest operation during installation.

Also, take a look at php::SplFileInfo->isDir(). It called 13'937'541 times, and almost all the calls (13'919'940) comes from Drupal\Core\Config\FileStorage->getAllCollectionNamesHelper(), but php::SplFileInfo->isDir() is slow by itself:

This is clearly because Drupal\Core\Config\FileStorage->getAllCollectionNamesHelper() doesn't cache its result in any way. It's always calculated in runtime. It seems to me like a bottleneck.

Steps to reproduce

Install the website from existing configs and profile it. The more configs you have, the more problems you will see.

Proposed resolution

Provide at least a static caching for the results of this method.

Remaining tasks

  1. Provide a patch with a static caching and see how tests are going.
  2. Decide should it be cached and how.
  3. Implement it.
πŸ“Œ Task
Status

Needs work

Version

11.0 πŸ”₯

Component
ConfigurationΒ  β†’

Last updated 1 day ago

Created by

πŸ‡·πŸ‡ΊRussia Niklan Russia, Perm

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @Niklan
  • Status changed to Needs review about 1 year ago
  • last update about 1 year ago
    30,195 pass, 5 fail
  • πŸ‡·πŸ‡ΊRussia Niklan Russia, Perm

    This is a very dirty patch to see how tests will react on it.

    This is TOP 10 calls during installation after fix applied.

    You can see that both, Drupal\Core\Config\FileStorage->getAllCollectionNamesHelper() and php::SplFileInfo->isDir() disappears from the list.

    • Drupal\Core\Config\FileStorage->getAllCollectionNamesHelper() calls went from 36'392 β†’ 4'551
    • php::SplFileInfo->isDir() calls went from 13'937'541 β†’ 23'710

    I measured installation time by time drush site:install --existing-config and results are:

    • Before: real 8m 38.06s
    • After: real 7m 9.09s

    It's roughly ~15% increase in site installation speed.

  • Status changed to Needs work about 1 year ago
  • πŸ‡·πŸ‡ΊRussia Niklan Russia, Perm

    It looks like some code will break. Can we decorate that service specifically for installation process? Because I don't think this caching is actually needed for regular runtime process, but it's a very significant improvement for installation. I don't see any cases where configurations are changed on disc during installation, hence, this caching should not do any harm.

  • πŸ‡«πŸ‡·France andypost

    Is it happening during installation only?

  • πŸ‡·πŸ‡ΊRussia Niklan Russia, Perm

    This profiling results for drush site:install --existing-config with a specific website configuration. It is not an issue on default profiles (because they are simple and small) or a smaller website (which have fewer configs). But a multilingual website, with something like 10 languages, a lot of content types, fields, etc., configs grows in size very fast, and this is where the issue starts. The profiler is clearly shows, that the more configurations you have, the more calls for Drupal\Core\Config\FileStorage->getAllCollectionNamesHelper() which is bottlenecked by a php::SplFileInfo->isDir() - the unique calls of which is hundreds times fewer than the actual one. In my example, it means for one unique file, it calls this check 587 times.

    This issue reproduced everywhere, on different developers PCs and CI's.

  • πŸ‡¬πŸ‡§United Kingdom alexpott πŸ‡ͺπŸ‡ΊπŸŒ

    I feel we need to fund out why we're call getAllCollectionNames() and work from there.

  • πŸ‡«πŸ‡·France andypost

    static cache is workaround, as file cache should be applied in other place

  • πŸ‡«πŸ‡·France andypost

    bottlenecked by a php::SplFileInfo->isDir() - the unique calls of which is hundreds times fewer than the actual one. In my example, it means for one unique file, it calls this check 587 times.

    it means filecache miss

Production build 0.71.5 2024