Make external entities translatable

Adding ✨ Translation support in vertical data aggregation Active as related.

@rp7: Would you kindly let us know what's left to do here? Thanks.

Been using this on a quite busy project in production for a few months now, for 3 separate annotatable external entity types. No issues reported so far.

But it now appears to be in conflict with ✨ Translation support in vertical data aggregation Active , which is tailored to a different kind of external API format.

Besides that, test coverage is non-existent yet.

Thanks for the update!

Okay, so we need to figure out how to incorporate both approaches without collision.

Updated title so I stop getting it confused with ✨ Translation support in vertical data aggregation Active .

With #3506455, we can (I did not succeed yet though...) merge several language sources into a single external entity with sub-arrays keyed by __external_entity_translation__<LANG_CODE>.
For instance, example 1:

[
  'id' => 42,
  'title' => 'English title',
  'somefield' => 'other text',
  'someotherfield' => 501,
  '__external_entity_translation__fr' => [
    'id' => 42,
    'title' => 'Titre français',
    'somefield' => 'autre texte',
    'someotherfield' => 501,
  ],
]

@rp7, in your current approach, you expect raw data to be an array keyed by languages.
For instance, example 2:

[
  'en' => [
    'id' => 42,
    'title' => 'English title',
    'somefield' => 'other text',
    'someotherfield' => 501,
  ],
  'fr' => [
    'id' => 42,
    'title' => 'Titre français',
    'somefield' => 'autre texte',
    'someotherfield' => 501,
  ],
]

What I don't like with example 1 is that untranslated fields can be duplicated (and it could cost a lot of memory, depending what is stored) and what I don't like in example 2 is that some object may be untranslated but have a key corresponding to a language code which may lead to an incorrect "language availability guessing". If you systematically change the raw structure of all external entities to add a language layer, then there would be no "incorrect guessing" but it would add a layer to raw data and make things more complicated.

I agree I don't see better/ideal solutions at the moment, but we should discuss how things could be handled here. From my side, my main concern is not how the final translatable raw structure will look like but rather how that structure will be generated.

For instance, you could consider consuming a REST API where the language is selected in the URL: you would use a vertical aggregator with as many source as languages you support. Each source provides values for a given language.
Now, if you consider a TSV file (Excel-like file) where there is a "lang_code" column telling the language of the record. You could find multiple time the same ID but with a different value for its "lang_code" column. How to aggregate that? Or should we consider each translated record as independent, but resolve the record to use using both its identifier and its lang_code? It can be interesting to do so to only load the appropriate translation.

So far, ideally, I would prefer the system to be able to only load the appropriate translation, without querying multiple sources (or multiple time the same source). For REST clients, it would mean to select the appropriate URL according to the selected language, for TSV clients or an SQL client to filter on a language column, for a file system client maybe to select the directory scheme according to a language code, etc. I think language should be managed by the storage clients because they know how to filter by language. Otherwise, they would always return all the translations and it would not be efficient. Now, how could that be achieved? I believe there should be a language parameter in the storage client ::load()/::loadMultiple() methods but it's an API change (fortunately, we're still in beta) or a service could be used to get the language instead? What if we want to edit and save another language while the current language is different? I need to think more about it but if you have ideas, please share! :)

What I would prefer to get is example 3:

• Query storage client to get entity 42 in "en":
  

  [
    'id' => 42,
    'title' => 'English title',
    'somefield' => 'other text',
    'someotherfield' => 501,
  ]
  

• Query storage client to get entity 42 in "fr":
  

  [
    'id' => 42,
    'title' => 'Titre français',
    'somefield' => 'autre texte',
    'someotherfield' => 501,
  ]
  

Here is my new proposal to manage languages:

It will be managed by ExternalEntityType class. On the external entity type edit form, there will be a new horizontal tab "Language settings", just below the "Storage" tab. On that tab, the user will be able to select a language and it will display current storage configuration form with language specific overrides. In other words, we would have a base storage config (data aggregator config) with possible overrides by languages. For instance, for a REST client, the REST service URLs could be overridden according to the language; for TSV clients, the source file could be changed or a filter could be added; for SQL clients, the queries could be adapted. Since it's the aggregator config that is overridden, it would also be possible to even change the storage client (ie. provide translations from another storage client). The idea would be to provide a checkbox to allow a "language" storage config override and highlight what is overridden (on client side through Javascript). Unchecking+re-checking the checkbox would reset "language" storage config for update. It will be possible to improve this user interface later.

On the run time, the config to load according to current language would be selected by the external entity type when ExternalEntityType::getDataAggregatorConfig() is called.

No change is required to storage clients (including base). No change is required to ExternalEntityStorage.
Modified: ExternalEntityType, ExternalEntityTypeForm and external entity type config schema (...and ExternalEntity for ->setTranslatable(TRUE) ;-) ).

That could be enough but we could get a step further to simplify user config management: we can take profit of the Token module recently added as a dependency to use it in storage clients where it could be pertinent. For instance, a REST service URL could use the token [language:langcode] and may not need to override storage config for any language.

What do you think?

Sounds reasonable.