Configuration language being overwritten during module install

Updated summary with remaining UI related 🐛 There is no indication on configuration forms if there are overridden values Needs work

and replaced fixed issue with 🐛 Prevent saving config entities when configuration overrides are applied Needs work

Hello everyone,

I attempted to reproduce this issue on my local environment, and I successfully replicated it by following these steps:

step 1) Freshly installed Drupal.
step 2) Installed essential multilingual modules such as locale, content_translation, config_translation, and language.
step 3) Added a new language, Dutch, through the admin interface (admin/config/regional/language),and added dutch as a default.
step 4) downloaded Dutch translations from localize.drupal.org.
step 5) Went to /admin/config/regional/translate/import and imported the translation file (.nl.po extension).
step 6) Examined the configuration of a content type (in this case, the Article content type) and confirmed that all values were stored in Dutch (see screenshot: article content type config.png).
step 7) Installed a module with default configurations; for example, I tried the Book module.
Step 8)After installing the Book module, I tested the configuration of the Book content type and noticed that it forcefully overrides the configuration language to Dutch (see screenshot: book content type configuration.png).

Upon investigation, I found that the issue is related to the locale_system_set_config_langcodes function in local.module. The function calls updateDefaultConfigLangcodes, which contains the following code:

$default_langcode = $this->languageManager->getDefaultLanguage()->getId(); //this will provide selected default language in site.
$langcode = $config->get('langcode'); // this will provide the configuration language like in book module have "en".
if (empty($langcode) || $langcode == 'en') {
   $config->set('langcode', $default_langcode)->save();
}

This code forcefully sets the configuration language to the default site selected language whenever a module or theme is installed
This behavior seems to be affecting the Book module, where the configuration language code is 'en' as actual language.

I have added a logger in tracing the execution flow and identifying the specific language code being added to the Book module configuration. Notably, the logger revealed that the Book module's configuration is assigned the language code 'en' (see screenshot: book config selected languagecode.png).

I have taken reference from the #38 , as it prevent the config overrides to default language to default config file of module. as i have tested this book module only after applying this path book module default config value is not overrides with dutch language its in English only.
i have added a patch against 11.x .
Please review.

i have added a MR with test. please review.

MR appears to have a test failure.

Added some nitpicky comments to the MR that should be addressed too while tests are updated.

Addressed the mentioned comment, please review.

I wonder if the test be a kernel test rather than a browser test? I'm going to experiment with converting it :)

With the current MR, going to admin/config/regional/language crashes with:

> Symfony\Component\DependencyInjection\Exception\ServiceCircularReferenceException: Circular reference detected for service "router.route_provider", path: "options_request_listener -> router.route_provider -> cache_tags.invalidator -> maintenance_mode_subscriber -> url_generator". in Drupal\Component\DependencyInjection\Container->get() (line 147 of /var/www/html/repos/drupal/core/lib/Drupal/Component/DependencyInjection/Container.php).

I can't work out how the translated node type label is supposed to show when you go to de/admin/structure/types/manage/page -- it doesn't work manually for me at all.

So I can't figure out how the actual test part of the test could be done using a kernel test, as I can't work out what's happening when it goes wrong. How does the translated node type label get loaded to be shown in the form?

In the meantime, I'm streamlining the Functional test with code from the kernel test I started writing.

I'm also removing the mentions of locale_test_translate -- that's not being enabled in this test, so it looks like frankencode comments to me :)

I wonder whether the kernel tests in LocaleConfigSubscriberForeignTest are relevant here -- they seem to be testing the same sort of things.

Ok I am now fairly sure that test coverage for this should be added to LocaleConfigSubscriberForeignTest. But I am very confused about how to make that work:

  public function testInstallModuleWithConfiguration() {
    // Install the Language module's configuration so we can use the
    // module_installer service.
    $this->installConfig(['language']);
    $this->container->get('module_installer')->install(['locale_test_translate']);
    $this->installConfig(['locale_test_translate']);


    // Do we need to do this?
    locale_system_set_config_langcodes();
    $langcodes = array_keys(\Drupal::languageManager()->getLanguages());
    $names = Locale::config()->getComponentNames();
    Locale::config()->updateConfigTranslations($names, $langcodes);

    // Not this -- it fails on both 11.x and the feature branch.
    $this->assertEquals('hu', \Drupal::service('locale.config_manager')->getDefaultConfigLangcode('locale_test_translate.settings'));

    // ???
    $this->assertEquals('hu', $this->configFactory->getEditable('locale_test_translate.settings')->get('langcode'));

My problem is that I don't understand the underlying architecture of how config is translated to understand what all the test helpers do and what the services like localeConfigManager / LocaleTranslation etc do. I suspect this is the case with most people and that the translation system has a very low bus factor!

I'm digging some more in this.

I'm confused by the fix in the MR being in code that says this:

      // Update active configuration copies of all prior shipped configuration if
      // they are still English. It is not enough to change configuration shipped
      // with the components just installed, because installing a component such
      // as views may bring in default configuration from prior components.

'prior shipped configuration' means config that was installed from a module *before* the current install operation. So I don't see how that's going to help with the bug, which is about something that goes wrong with config from a module *as it's being installed*.

But then I'm not sure what updateDefaultConfigLangcodes() is trying to do anyway -- what does 'default' mean in this context? In getDefaultConfigLangcode(), 'default' means 'shipped with a module's code in config/install'. But here we're updating config, so it's not the shipped version is it?

Whenever a module with configuration settings is installed, the configuration is consistently overridden by the default language of the site if it is not set to English. The updateDefaultConfigLangcodes function is tasked with updating the language code of the configuration provided by that module. To mitigate this issue, I have implemented a check to verify whether the language code of the current configuration contains 'en' (English) and if 'en' is already included in the available language codes. This check prevents the configuration from being overridden by the default selected language.

It would be really good if the code comment explained that a bit more, and more precisely, explained the circumstances under which the language code should be updated.

I think this logic could be made clearer:

          $langcode = $config->get('langcode');
          $is_available_langcode = in_array($langcode, $available_langcodes);
          if ((empty($langcode) || $langcode == 'en') && !$is_available_langcode) {

In the case that $langcode is empty, then it's obviously not in the array. So we only need to check if it's $available_langcodes if it's 'en', AFAICT?

So for example, this would be clearer:

          if (empty($langcode) || ($langcode == 'en') && !$is_available_langcode) {

And while I am generally REALLY in favour of splitting up complex conditionals, I'm not sure breaking out the check for $is_available_langcode is good for readability here, and it's not efficient, as it's checking even if $langcode is empty, or not 'en'.

I'm still not sure how we test the patch using the API.

I've installed book when 'fr' is the default language, with and without the fix, and looked at the {config} table.

Without fix:

	node.type.book	a:12:{s:4:"uuid";s:36:"10752ba3-9235-4540-a959-c9e9190376ae";s:8:"langcode";s:2:"fr";s:6:"status";b:1;s:12:"dependencies";a:1:{s:8:"enforced";a:1:{s:6:"module";a:1:{i:0;s:4:"book";}}}s:5:"_core";a:1:{s:19:"default_config_hash";s:43:"xvVZ9piiDxh4ziNyl-YqCT5vF8nI7xyupTdQWlN-Hxk";}s:4:"name";s:9:"Book page";s:4:"type";s:4:"book";s:11:"description";s:87:"<em>Books</em> have a built-in hierarchical navigation. Use for handbooks or tutorials.";s:4:"help";N;s:12:"new_revision";b:1;s:12:"preview_mode";i:1;s:17:"display_submitted";b:1;}
language.en	node.type.book	a:2:{s:4:"name";s:9:"Book page";s:11:"description";s:87:"<em>Books</em> have a built-in hierarchical navigation. Use for handbooks or tutorials.";}

With fix:

	node.type.book	a:12:{s:4:"uuid";s:36:"88e05137-2a0e-4a18-840e-60769f3dccd7";s:8:"langcode";s:2:"en";s:6:"status";b:1;s:12:"dependencies";a:1:{s:8:"enforced";a:1:{s:6:"module";a:1:{i:0;s:4:"book";}}}s:5:"_core";a:1:{s:19:"default_config_hash";s:43:"xvVZ9piiDxh4ziNyl-YqCT5vF8nI7xyupTdQWlN-Hxk";}s:4:"name";s:9:"Book page";s:4:"type";s:4:"book";s:11:"description";s:87:"<em>Books</em> have a built-in hierarchical navigation. Use for handbooks or tutorials.";s:4:"help";N;s:12:"new_revision";b:1;s:12:"preview_mode";i:1;s:17:"display_submitted";b:1;}

I can see there's a difference, obviously -- only one row rather than two!

But shouldn't I be seeing translated text in there?

I also don't understand what the API reports.

I've got this debug code:

$config_name = 'node.type.book';
      // enabled but not default language
      $override = $this->languageManager->getLanguageConfigOverride('en', $config_name);
      dump($override->isNew());

      // the default language
      $override = $this->languageManager->getLanguageConfigOverride('fr', $config_name);
      dump($override->isNew());

WITH the fix, I get FALSE, TRUE

WITHOUT the fix, I get TRUE, TRUE

I don't understand how BOTH are reporting they're new without the fix.

I think the whole config translation system should be simplified and could be solved if:
- "default" config (the yml config files shipped with modules), is always in langcode "en" and contains English strings (obviously).
- All module translations are loaded and imported before the config is imported, so string translation is available.
- For each default English config, a language override is created when installing a site in a different language or when additional languages are added. So a site with languages NL and FR, with FR the default language, has English configs and two overridden configs in Dutch and French.
That way you can switch default site language whenever you want, as much as you want, add languages, delete them. Without default config ever needed to be updated.

Heavy +1 for what @kriboogh said in #72. It's basically what I also said in #2806009-51: Installing a module causes translations to be overwritten → . That issue was fixed to address the symptoms at hand, but the root cause was never really addressed for all I can tell.

We require all projects on DO to ship their config in English, so why don't we make use of that and keep the config/sync folder English at all times? This would make everything so much easier from a code perspective too as we then know that the main config folder is always English and the language-specific folders are always for said language.

I also agree with #72 / #73. Thanks!

We require all projects on DO to ship their config in English, so why don't we make use of that and keep the config/sync folder English at all times?

I was trying to think what the effect of this would be on a monolingual non-English site. I think in that case, if you really, really didn't want to enter English for some config, you could just add it in whatever language (let's say French), the system would store your French text in config/sync as 'English', and then you'd have to translate from 'English' to French (copy and paste the same thing or leave it untranslated).

And even though that would be a bit of a workaround, it seems like it'd be really predictable what would happen, so still OK overall. And then all the other situations would be simplified and make a lot more sense. I only deal with one multilingual site and even that site is not fully multilingual, but with that caveat, +1 from me too.

+1 for default config always being in English.

So what would be the next steps to fix this? I've found a handful of issues that all boil down to the same root cause described in #72 / #73. If we want to switch to a way where the main config folder is always EN, which steps can we already take without breaking BC or with a solid update path?

What I'd like to see here would be a new entry to the system.site.yml file that would have the following value:
config_langcode: en
This would be set to English if the site is installed in English, or to whatever other language is the site's install language.

At the moment, there would be no provision to change this value in the UI, but if we ever figure out a way to do that, while not messing up a lot of translations, that could change in the future.

I think that by allowing the config_langcode to be configurable like that, you're gonna end up in the same mess if someone changes that config will the site already has config in the first language.

By having a default config in EN always and have the language config overrides system (which already is in place and working) deal with the translations, everything should basically work out of the box (sort of). The only problem is "fixing" existing websites when we introduce this fixed EN config langcode.

We have done this in a current project. Basically you need to get all default config that was stored in the current default language (for example NL). Change the langcode of the default config to EN. WE now have a EN default config with NL data. So look if a EN config override exists. If there is, swap all data from the EN override into the new EN default. If there isn't an EN override, look for the original EN source strings of the data in the NL data and copy that into the new default EN config. To create a NL override, we can either let core create it (by looking up the translations through locale, or copy the old default translatable NL keys (labels, text,...) from the original NL default config into the new NL override.

We have done this in a current project. Basically you need to get all default config that was stored in the current default language (for example NL). Change the langcode of the default config to EN. WE now have a EN default config with NL data. So look if a EN config override exists. If there is, swap all data from the EN override into the new EN default. If there isn't an EN override, look for the original EN source strings of the data in the NL data and copy that into the new default EN config. To create a NL override, we can either let core create it (by looking up the translations through locale, or copy the old default translatable NL keys (labels, text,...) from the original NL default config into the new NL override.

This seems sufficiently complex that we might not be able to provide an update hook for it in core.

I'm not sure exactly how (maybe a flag in settings or a container parameter), but can we look at forcing English for default config on new installs, which would fix the bug for new sites, and then we could at least provide instructions (and maybe a contrib module?) to help existing sites convert based on the above?

If we force it on new sites only, then the code base also needs to take care of both scenarios: old and new. I can already sense the amount of bug reports we'd be getting then. If we provide an update hook, then the code base only needs to take the new scenario into account.

@kriboogh would you mind sharing your update code here? Perhaps it might be simpler than it sounds.

@kristiaanvandeneynde sure, it's not pretty though :D
We had to do some specific stuff with system.site config, which might have been only related to our project. But I'll keep it in.

Disclaimer: To anyone else reading this, this is specific project code given as an example, do not run this blindly on your production site !!!

function my_module_update_9000(&$sandbox) {

  // See patch https://www.drupal.org/project/drupal/issues/3150540.
  \Drupal::configFactory()->getEditable('locale.settings')->set('update_default_config_langcodes', FALSE)->save();

  // 1. Fix system.site config.
  $db = \Drupal::database();
  $db->delete('config')
    ->condition('collection', 'language.en')
    ->condition('name', 'system.site')
    ->execute();

  // Make sure the default config is set up as 'en'.
  $default_langcode = \Drupal::languageManager()->getDefaultLanguage()->getId();
  $default_config = \Drupal::configFactory()->getEditable('system.site');
  if ($default_config->get('langcode') == $default_langcode) {

    // Keep the default langcode name, we need it for the new override.
    $old_default_config_data = $default_config->getRawData();

    // Change the default langcode to english.
    $default_config->set('langcode', 'en');

    // Check if we have an english translation override, copy over its values.
    $config_en_translation = \Drupal::languageManager()->getLanguageConfigOverride('en', 'system.site');
    if (!$config_en_translation->isNew()) {
      $default_config->set('name', $config_en_translation->get('name'));
      $default_config->set('mail', $config_en_translation->get('mail'));
    }
    $default_config->save();

    // Create a new language override for the default langcode and store
    // the old values.
    if (!empty($old_default_config_data['name']) || !empty($old_default_config_data['mail'])) {
      $config_default_langcode_translation = \Drupal::languageManager()->getLanguageConfigOverride($default_langcode, 'system.site');
      if (!empty($old_default_config_data['name'])) {
        $config_default_langcode_translation->set('name', $old_default_config_data['name'] ?? '');
      }
      if (!empty($old_default_config_data['mail'])) {
        $config_default_langcode_translation->set('mail', $old_default_config_data['mail'] ?? '');
      }
      $config_default_langcode_translation->save();
    }
  }

  // Now cleanup the language overrides, they should only contain the site name and mail.
  foreach (\Drupal::languageManager()->getLanguages() as $langcode => $language) {
    if ($default_config->get('langcode') != $langcode) {
      // Only do overrides, skip the default config.
      $config_translation = \Drupal::languageManager()->getLanguageConfigOverride($langcode, 'system.site');
      if (!$config_translation->isNew()) {
        // Ignore non existing overrides.
        $config_translation->clear('page');
        $config_translation->save();
      }
    }
  }

  // 2. Check all default config langcodes and check if a language override
  // exists with the same langcode. Fix that.
  $query_string = <<<MYSQL
    SELECT c1.name, c1.collection
    FROM config c1, (SELECT c.name,
        REGEXP_REPLACE(REGEXP_SUBSTR(c.data, 's:8:"langcode";s:2:"(..)"'), '.*s:2:"(..)"', '\\\\1') as langcode
        FROM config c
        WHERE
          c.collection = ''
        AND
          c.data REGEXP 's:8:"langcode";s:2:".."'
        ) as clang
    WHERE
    c1.name = clang.name
    AND
    c1.collection = CONCAT('language.', clang.langcode)
  MYSQL;

  $rows = $db->query($query_string, [], ['allow_delimiter_in_query' => TRUE])->fetchAllAssoc('name');
  foreach ($rows as $row) {
    $db->delete('config')
      ->condition('collection', $row->collection)
      ->condition('name', $row->name)
      ->execute();
  }

  // 3. Swap default language config with language.en overrides.
  $query_string = <<<MYSQL
    SELECT c1.*
    FROM config c1, (SELECT c2.name FROM config c2 WHERE c2.collection = '' AND c2.data REGEXP 's:8:"langcode";s:2:"{$default_langcode}"') as c2
    WHERE c1.name = c2.name and c1.collection = 'language.en'
  MYSQL;

  $rows = $db->query($query_string, [], ['allow_delimiter_in_query' => TRUE])->fetchAllAssoc('name');
  foreach ($rows as $name => $row) {

    $default_config = \Drupal::configFactory()->getEditable($name);
    $english_config_override = \Drupal::languageManager()->getLanguageConfigOverride('en', $name);

    if (!$english_config_override->isNew() && !$default_config->isNew()) {
      // Only do this if both configs already exist!
      // Copy the english override into the default config and change it
      // to english.
      $english_config_override_data = $english_config_override->get();
      $default_config_data = $default_config->getRawData();
      $english_default_data = array_replace_recursive($default_config_data, $english_config_override_data);
      $default_config->setData($english_default_data);
      $default_config->set('langcode', 'en');
      $default_config->save();

      // Now take the original default config values and create a new
      // language override with the default language. Store only the keys
      // from the default config that where present in the english override.
      $default_config_override = \Drupal::languageManager()->getLanguageConfigOverride($default_langcode, $name);
      $default_config_override_data = array_intersect_key_recursive_values_2($english_config_override_data, $default_config_data);
      $default_config_override->setData($default_config_override_data);
      $default_config_override->save();

      // Delete the english override.
      $english_config_override->delete();
    }
  }

}

/**
 * Do a recursive key intersect but return values from array 2.
 *
 * @param array $array1
 * @param array $array2
 *
 * @return array
 */
function array_intersect_key_recursive_values_2(array $array1, array $array2) : array {

  $intersection = [];

  foreach ($array1 as $key => $value) {
    if (isset($array2[$key])) {
      if (is_array($array1[$key]) && is_array($array2[$key])) {
        $intersection[$key] = array_intersect_key_recursive_values_2($array1[$key], $array2[$key]);
      }
      else {
        $intersection[$key] = $array2[$key];
      }
    }
  }

  return($intersection);
}