Problem/Motivation
Since v3 is not stable yet, we still have room for major changes and I'd like to discuss one here. Current "philosophy" of external entities is to have a storage client extracting raw data from a source and a field mapper to adapt that raw data to Drupal. With v3, I've introduced another layer: the data aggregator which allows to aggregate data from multiple sources into one dataset.
One problem I met is that it is quite common to have raw fields that need specific formatting while most of the other can be used with a simple "direct" field mapping. That's why I modified the field mapper user interface to allow the user to select a type of mapping for each mapped field. It became possible to use the JSONPath field mapper but use direct field mapping in some cases instead of always using a JSONPath expression. It also became clearer when constant values where used (not more special notation with a "+" that could be confusing for new users).
I would like to go a step further and enable the selection of a field mapper on a field basis rather than on an external entity type basis. A good starting example is
#2999505: Improve date handling →
. When you need to map a special raw data field, for instance a date field or a GPS coordinate field, or a more complex structure, one single field mapper plugin may not be flexible enough to handle all those case if they all appear in a given external entity type.
Proposed resolution
The previous concept of field mapper changes: now, a field mapper does not
handle all the fields abd their properties anymore.
The external entity handler (type) provides the list of mappable fields.
For each field a field mapper can be choosen. A field mapper can support a
restricted list of field types: for instance, there can be a dedicated field
mapper for formated text fields that provide a dedicated (constant) mapping for
text formats, a dedicated field mapper for file/image field types that could
handle specificaly file identifiers, one for geo-locations, etc. But there
would also be a generic field mapper that would support any field type.
Each field mapper would enable to map field properties using a property mapper
for each property that needs a mapping. The property mappers would be very
similar to previous field mappers: their purpose would be to map a property
value to a value extracted from source data.
There will be a "direct property mapper" mapping from a source field to a
property, a "simple property mapper" that could map source subfields to a
property, a "constant property mapper" that can fill properties with
user-provided constant values, a "JSONPath property mapper", etc.
Each property mapper has a method that tells if it can reverse the mapping or
not and a method to add reversed value from an external entity to a raw data
array.
Property mappers can and should use a new type of plugin: data processors. Data
processors get a data structure and transform it into a new structure. Typical
examples of data processors would be date and time format converters, numeric
value extractors (that would remove units for instance), numeric value
normalizers that would convert units (eg. inch to cm, or dm/cm/mm to m), value
mappers that would associate a value to another, translators, hash algorithms,
etc. Similarly to property mappers, data processor also have a method that tells
if it can reverse the process to restore an original value and a method that
does it if possible. A property mappers that allows reverse mapping will also
have to ensure the data processor it uses also allows reverse processing to tell
if data can really be reverse-mapped (ie. it is not the field mapper
responsability to check both since the field mapper does not know if the
property mapper uses a data processor nor which one).
By default, each field would use a generic field mapper which itself would
enable the use of any available property mapper on every field property. By
default, property mapper would not use a data processor but allow to select one.
To regenerate a source data structure from an external entity, each field
mapper will call methods form its property mappers that allow reverse mapping
and provide the raw data structure being compiled and a context object that may
contain extra-data that can be shared between field mappers and property mapper
instances. For instance, a JSONPath object that parsed the external entity raw
data structure can be shared that way between multiple JSONPath property
mappers.
Remaining tasks
Gather input and ideas from the community and then start changing things...
User interface changes
To be determined.
API changes
To be determined.
Data model changes
To be determined.