locale_config_batch_update_components() is prohibitively slow with large amounts of config or languages

Created on 13 July 2017, over 7 years ago
Updated 11 October 2022, about 2 years ago

locale_config_batch_update_components() is very time-consuming on installs with large amounts of config and/or languages. I'd like to discuss if it is possible to move the batches created by this function into queue.

Code flow:

  • locale_modules_installed() and locale_themes_installed() both call locale_system_update()
  • locale_system_update(), after setting up batches to retrieve translations for the installed modules/themes, calls locale_config_batch_update_components()
  • locale_config_batch_update_components() sets up all known configuration translation to be reimported as batches.
  • Because of this flow, in a case where say multiple update hooks install modules, locale_modules_installed() gets called multiple time, which means all locale configuration updates are run multiple times as well.

In a build I work with, we have 1159 items returned by LocaleConfigManager()->getComponentNames(), and 29 active non-source languages. This makes 33611 items that need to be imported by batch. Importing them by batch means among other things that a Drupal database update is held open while these batches process.

Batching these items prevents PHP timeout, but other timeouts come into play. The amount of time a database connection is kept open while processing these batches will often exceed MySQL wait_timeout limits, for example.

I would appreciate input from people more familiar with the Locale system than me:

  • Would it be possible to push these batch items into queue instead, so that they can be executed properly in the background, without otherwise blocking progress?
  • In the case where there are multiple invocations of a module install, can we find a way to consolidate these batch sets into one batch? This can be taken care of by careful structuring/combining of update hooks, in some cases, but it is not always possible to consolidate multiple update hooks into one to avoid the multiple batching problem.

If a queue approach makes sense, I'm happy to supply a patch; but I'm concerned that there may be implications to not having all config updated immediately.

It looks like this issue was discussed and worked on some in https://www.drupal.org/node/2029075 β†’ , but I'm not sure the solution there properly solves the problem.

πŸ› Bug report
Status

Active

Version

11.0 πŸ”₯

Component
Language systemΒ  β†’

Last updated 1 day ago

  • Maintained by
  • πŸ‡©πŸ‡ͺGermany @sun
Created by

πŸ‡ΊπŸ‡ΈUnited States timcosgrove

Live updates comments and jobs are added and updated live.
  • Performance

    It affects performance. It is often combined with the Needs profiling tag.

  • needs profiling

    It may affect performance, and thus requires in-depth technical reviews and profiling.

Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • πŸ‡§πŸ‡¬Bulgaria yivanov

    This issue is still a problem with Drupal 10.2.2

    WIth a website that has 106 languages enabled and 392 configs returned by Locale::config()->getComponentNames(), we have following results when enabling modules:

    $ time drush en field_ui views_ui -y
    
     [success] Successfully enabled: field_ui, views_ui
    
    real    3m35.569s
    
    
Production build 0.71.5 2024