Problem/Motivation
Since v3 is not stable yet, we still have room for major changes and I'd like to discuss one here. Current "philosophy" of external entities is to have a storage client extracting raw data from a source and a field mapper to adapt that raw data to Drupal. With v3, I've introduced another layer: the data aggregator which allows to aggregate data from multiple sources into one dataset.
One problem I met is that it is quite common to have raw fields that need specific formatting while most of the other can be used with a simple "direct" field mapping. That's why I modified the field mapper user interface to allow the user to select a type of mapping for each mapped field. It became possible to use the JSONPath field mapper but use direct field mapping in some cases instead of always using a JSONPath expression. It also became clearer when constant values where used (not more special notation with a "+" that could be confusing for new users).
I would like to go a step further and enable the selection of a field mapper on a field basis rather than on an external entity type basis. A good starting example is
#2999505: Improve date handling →
. When you need to map a special raw data field, for instance a date field or a GPS coordinate field, or a more complex structure, one single field mapper plugin may not be flexible enough to handle all those case if they all appear in a given external entity type.
Proposed resolution
The previous concept of field mapper changes: now, a field mapper does not handle all the fields and their properties anymore.
The external entity handler (ie. external entity type) provides the list of "mappable" fields. For each field a field mapper can be chosen. A field mapper can support a restricted list of field types: for instance, there can be a dedicated field mapper for formatted text fields that provide a dedicated (constant) mapping for text formats, a dedicated field mapper for file/image fields that could handle specifically file identifiers (current companion module xntt file field), one for geolocations, etc. But there would also be a generic field mapper that would support any field type.
Each field mapper would enable to map field properties using a property mapper for each property that needs a mapping. The property mappers would be very similar to previous field mappers: their purpose would be to map a property value to a value extracted from source data.
There will be a "direct property mapper" mapping from a source field to a property, a "simple property mapper" that could map source sub-fields to a property, a "constant property mapper" that can fill properties with user-provided constant values, a "JSONPath property mapper", etc.
Each property mapper has a method that tells if it can reverse the mapping or not and a method to add reversed value from an external entity to a raw data array.
Property mappers can and should use a new type of plugin: data processors (the concept of field processor developed in
External Entities Field Processors →
). Data processors get a data structure (string|array) and transform it into a new structure. Typical examples of data processors would be date and time format converters, numeric value extractors (that would remove unit characters from a value for instance), numeric value "normalizer" that would convert units (eg. inch to cm, or dm/cm/mm to m), value mappers that would associate a value to another, translators, hash algorithms, etc. Similarly to property mappers, data processor also have a method that tells if it can reverse the process to restore an original value and a method that does it if possible. A property mappers that allows reverse mapping will also have to ensure the data processor it uses also allows reverse processing to tell if data can really be reverse-mapped (ie. it is not the field mapper responsibility to check both since the field mapper does not know if the property mapper uses a data processor nor which one).
By default, each field would use a generic field mapper which itself would enable the use of any available property mapper on every field property. By default, property mapper would not use a data processor but allow to select one. To regenerate a source data structure from an external entity, each field mapper will call methods form its property mappers that allow reverse mapping and provide the raw data structure being compiled and a context object that may contain extra-data that can be shared between field mappers and property mapper instances. For instance, a JSONPath object that parsed the external entity raw data structure can be shared that way between multiple JSONPath property mappers.
Remaining tasks
Implement.
User interface changes
The field mapping page changes: there will be a field set for each external entity field. In that field set, there will be a drop-down allowing to select a compatible field mapper" (a generic field mapper will always be available - maybe excepted for the id field?). A field mapper would provide a user interface to map properties when needed. Some properties might be hidden to user, some other might have dedicated options and some other will use property mappers that can be chosen using a drop-down. A property mapper provides a user interface to map raw source fields to a Drupal property. It would typically be a text field that would hold an expression and it might not be a "one-to-one" mapping and even not a reversible mapping. Additionally, a property mapper can allow data alteration of the fetched or generated value using data processor. A data processor could be selected using a drop-down as well and may or may not provide additional settings. For instance, a data processor that just extract a numeric value may not need any additional settings while a unit converted might need to know the target unit and would provide something to select it.
API changes
All the field mapper API changes as it is now split into field mappers + property mappers + data processors.
Data model changes
The config of external entity type will change to support those new settings.