Make recipes translatable

At least YAML 1.2 allows for tags that allow us to define custom data types https://yaml.org/spec/1.2.2/#tags, then we would need to respond to the !translate type with wrapping it in translation at runtime and extracting it statically. Not sure how wide support for this is in tools.

key: 12
another: !translate This text will be translated

The default (failsafe) schema of YAML does not include !translate, however

All nodes with the “!” non-specific tag are resolved, by the standard convention, to “tag:yaml.org,2002:seq”, “tag:yaml.org,2002:map” or “tag:yaml.org,2002:str”, according to their kind.

per https://yaml.org/spec/1.2.2/#failsafe-schema -- which I understand that tools are expected to resolve custom tags as sequence, map or string based on which syntax was used above them. So !translate followed by a string would/should be understood as string when applied to a string by a YAML tool following the failsafe schema.

Adding related issues.

We'll need to figure out too:

* Potx support
* l.d.org support: we need to run potx on general projects that include a recipe.yml.

Any news here?

Looks like the Symfony YAML parser supports custom tags: https://symfony.com/doc/current/components/yaml.html#parsing-and-dumping...

Maybe something we could take advantage of here?

About having "recipes" as a thing - meaning the recipe's yaml strings - being translatable, can't we just use the same approach used for module's yml files? - I mean something like adding the files here.. (potx module) https://git.drupalcode.org/project/potx/-/blob/8.x-1.x/yaml_translation_...

Then the other big issue is having the content included in the recipes - Default content? - translated which may be a whole different story - and issue.

I think the pieces of the recipe where translatable pieces may appear are not predictable like that? Which is why I suggested !translate in #3 above. Or do you see a way we can predictably say which parts will be translatable? I think they would depend on whether the config they reference for example are translatable, which I'm not sure we can resolve?

The content is already translatable. Default Content exports translations and core imports them, and I'm pretty sure we have test coverage of that.

@gábor hojtsy
Not sure about which parts need to be translated, just trying to identify the pieces of this issue and noting this should be somehow part of how Drupal handles translation for generic yml files, though these ones may be more complex..

@phenaproxima,
You're right, default content is translatable. What we are missing though is some workflow to be able to translate that default content and include those translations with the CMS distribution. Same happens for Drupal's Umami profile, which includes English and Spanish in the download.

So we need to either keep this default content editable / translatable somewhere online, and then include it in the downloadable package, for every language, which I don't think is easy/doable with current infrastructure, or..

... maybe find some way for translatable strings in content to be extracted and available on LDO for translation, then downloaded on the fly, kind of like configuration translations... or...

... extend LDO to translate texts in default content, make the translated content downloadable... or....

well, I don't know, just pointing out the issue and that we need some new and fresh approach for this.

Being able to easily edit the translations is a separate problem from being able to translate them I think. Let's try to focus on translating recipes here?

Maybe symphonies approach can help here?
https://symfony.com/doc/current/translation.html

They have a way to handle yml files according to their docs as well and Drupal is based on it, isn't it?

I have to agree with @gàbor #13 about focusing on the recipes stuff first, not default content for now.

About extending yaml format / parsers, though it may open some interesting options, I don't think that is really what we need.

I've been playing around with strings, recipes, configurations and translations and as I see the problem, besides translating the strings in recipe.yml which looks like a trivial one, see ✨ Extract strings from Recipes Active translating the configuration shipped with recipes looks pretty much like the issue with module's configuration, only the yml files - and the schemas - are organized differently.

The main feature for having all these - modules, themes, recipes? - translated is being able to extract the translatable strings, for which we need basically to locate the configuration files and the configuration schema and that is the potx module, right? However, I've been running some tests and it looks like for individual modules, it also fails when extracting the strings for configuration whose schema is not in the module itself - still not sure whether having the module including the schema around will help.

When editing / installing the Recipe the modules providing the configuration schema are supposed to be available / installed so configuration translation will work just fine provided we have the translations available. So the only piece that is left is being able to parse the recipe package by itself and extract the translatable strings and that must be done by the potx or similar module.

I'm thinking two options right now that could also help with modules' strings: one is having the schema files located and loaded dynamically which may get really complex, and the other is having the Recipe / module providing some po file with these extra strings - that could be also generated by potx extended module but with the module and all the dependencies installed.

1. I think ✨ Extract strings from Recipes Active is a good one and simple one to make the labels of the recipes itself translatable.

2. Also config shipped with the recipes should be translatable (is it already?).

3. But finally what's not yet being solved and where I suggested the in-place YAML format support is config action values, eg. if you have a config action that updates the title of the a menu item or a views display etc that will save as-is and not translated.

2. Also config shipped with the recipes should be translatable (is it already?).

Yes, it should. AFAIK the missing piece here is the strings not getting extracted by potx.

3. ... where I suggested the in-place YAML format support is config action values ...

Ok, this is the part I was missing. Yes, I think these YAML tags look like a good option.

>> 2. Also config shipped with the recipes should be translatable (is it already?).

> Yes, it should. AFAIK the missing piece here is the strings not getting extracted by potx.

I wouldn't expect changes needed at potx for shipped config, but we probably need l.d.o to parse general projects and check if they are recipes looking at composer.json?

Today, I met with the multilingual track leads for Drupal CMS, and we determined that this issue -- with all its extensive implications -- is our main technical blocker.

I'm new to multilingual stuff (I live in 'Murica and therefore am only fluent in one language), but I'm eager to learn. Please forgive any ignorance on my part. :)

From my cursory research, I see four distinct phases emerging here:

1. We need to make sure all translatable strings are extracted from recipes. That includes all translatable strings in the recipe's config, the translatable values used in config actions, and the translatable metadata in recipe.yml. Ensuring that all of these things are extracted properly is clearly the foundation of everything else to come, and it looks like @jose reyero has already been making some headway here.
2. Once we have all the necessary strings extracted, we need to ensure they get onto localize.drupal.org. I don't know how this works, but I assume it will involve @drumm making some infrastructure changes.
3. Once a particular recipe has translations available on localize.drupal.org, we need some way to make those translations accessible to the recipe runner. We could change the recipe system so it tries to download the translations before applying the recipes -- like the way it handles modules, if my understanding is correct -- but I think that might prove quite fraught, since recipes are very composable and might be built on a bigger stack comprised of dozens of other recipes. I currently lean towards creating a Composer plugin that downloads a recipe's translations in the local filesystem at the same time as the recipe itself is installed by Composer, but that's merely one possibility; I'm certainly open to other options.
4. Finally, the recipe runner has to be sure that translations are properly applied! I'm not sure exactly how this will work, but it seems pretty certain that the underlying infrastructure for this already exists in core, so I imagine that this part of it would be reasonably straightforward.

I'd like to propose we turn this issue into a meta, organized around these four phases, with specific lists of issues that need to be fixed for each phase to be complete.

I think the plan in #19 looks great, adding the 'META' tag to this one for now.

And some patch about the approach discussed here using symfony tags, on this one https://www.drupal.org/project/drupal/issues/3313863#comment-16087199 ✨ Translation of config actions Active

I understand the need to make the strings in config actions something that can be translated, but the translation itself needs to apply to the config on the Drupal site itself- since it will be the config translation module that eventually does the actual translation in Drupal.

E.g. if you created that config by hand and translated it by hand, it should end up in the same format as if it was translated from shipped config no?

Which makes me wonder if there could be a build step on l.d.o, or for recipes themselves, that installs the recipe and exports the resultant config, so it can then be parsed for translation.

Hi @catch,

> ... the translation itself needs to apply to the config on the Drupal site
Right, but since the config translation goes through the localization system, or main issue is being able to track and export these locale strings - and for that these yaml tags are a handy solution.

Once these strings are into the locale system, they just need to be applied - on the fly, with that small patch - to config-actions strings. For regular configuration objects, the existing API should just work - though we need to make sure Recipes use it (?).

> ...a build step on l.d.o, or for recipes themselves, that installs the recipe and exports the resultant config...

Well, that could be an interesting alternate approach, though it looks to me way more complex and also it may not allow for runtime translation of config actions strings before applying them, I mean something like:

1. With tagged strings:
Existing config + Config action (translate string on the fly) -> Resulting config (translated). OK

2. With building step, we'll have the strings and their translations, but still cannot tell about the strings in the config actions.
Existing config + Config action (untranslated string) -> Resulting config (untranslated) -> Can we translate that?? (Not sure)

Putting it another way, our current config translation relies on being able to tell which strings are translatable within the shipped configuration and then translate those through the locale system. But with configuration actions we are adding some more strings on top of already existing - but not shipped with modules - site config, for which current translation/localization system may not work.

(And on top of that I'm afraid at some point we will need to think about applying recipes/actions to multiple language versions of the configuration...)

Which makes me wonder if there could be a build step on l.d.o, or for recipes themselves, that installs the recipe and exports the resultant config, so it can then be parsed for translation.

For l.d.o: I think this would result in a much bigger set of translatable strings. I'm not sure how we can tell apart from the resulting config which parts were from config actions and which ones were not? So far for all other string extractors we do get the specific strings, not a superset that includes a bunch of unrelated stuff :)

For a specific website: As Jose wrote, when recipes get applied on an actual site, they may get applied to a multilingual site where the default is not English anymore or the edited config value is not as expected. Let's consider this theoretic situation for the sake of demonstration:

• The default Drupal setting for anonymous username is "Anonymous"
• We have a recipe that changes that to "Unknown name"
• The site the recipe is applied on is in Spanish by default and changed this to "Sin nombre" manually.

In this situation, the config translation's shipped by Drupal for the username are for "Anonymous". Those are not useful as the site changed the setting.

The new value shipped with the config action is not the same as the Drupal default, but it is in English.

So when it is applied, the default config on the site needs the Spanish translation of "Unknown name" set. This requires recipe application to be aware that the value is translatable and needs to t("Unknown name") as part of the recipe application, which will possibly make "Nombre desconocido" the anonymous name.

If there was an English language config translation, THAT would get "Unkown name" applied from the config action at the same time and other translations would get the t("Unknown name") too.

Lifecycle after recipe applied: After the recipe is applied, there will be no trace that the recipe applied the use of "Unknown name" and that is somehow a source string that needs translations applied. We discussed that this is by design, so the config system cannot deal with translating "Unknown name" after the fact that the recipe got applied, because "Unknown name" does not even necessarily appear in config anymore and also because config does not know that "Unknown name" came from some shipped source code that could be t()-ed.

---

All in all I think the above point to needing to explicitly mark translatable strings in recipes, parse them out on l.d.o explicitly and t() them when applied, because afterwards there is no chance to do it.

All in all I think the above point to needing to explicitly mark translatable strings in recipes, parse them out on l.d.o explicitly and t() them when applied, because afterwards there is no chance to do it.

I might be missing something, don't think I've actually worked on a site using config translation yet.

If a site has English default language + French and Spanish config translations, how do both of the French and Spanish translations get into the language configuration override collection if we t() when the recipe is applied?

The config update and override creation/update needs to happen as part of the recipe application, before the recipe application is completed. After that the system has no idea if the "customized" string came from a source code (and can be t()-ed) or was manually entered (and cannot be t()-ed).

If a site has English default language + French and Spanish config translations, how do both of the French and Spanish translations get into the language configuration override collection if we t() when the recipe is applied?

We would t() the string from the config action to create or update the French and Spanish config overrides as part of the recipe application. Since the English is default, we'll not t() the config action string for the default language in this case. If the site has a Spanish default with English and French translations, then we would need to apply the untranslated config action string to the English translation on the site and t() for Spanish default config and t() for the French override.

---

When the recipe is applied, I guess one backward way of figuring out which config actions may be translatable is to backreference from the config schema as part of applying the recipe, but such backreferencing needs to be built into the recipe application then. However that still does not solve the l.d.o extraction problem. L.d.o does not have a Drupal runtime for extracting strings where the code would actually be run, it always statically inspects the code. (From a Drupal 7 system even at this time). It does not run the code it handles.

Thanks for #25, that sounds like it should work in terms of the recipe application side of things.

I'll assume that was in reference to the explanation above the "---", because under that I intended to outline a solution that is IMHO not really useful :)

Yes sorry the last paragraph gave me a headache so I skipped over it ;) but everything up to that answered my question and sounded sensible.