Aggregation by group prefixes

Created on 26 June 2023, over 1 year ago
Updated 2 August 2023, over 1 year ago

Problem/Motivation

The current aggregation system with "Aggregate all", "Aggregate by group prefixes with unprefixed storage clients in common" and "Aggregate by group prefixes and treate unprefixed storage clients as a default group" has design issues. With the expected behavior, it is not possible to count entities and properly use filters. The group prefix stripping function presents some bugs. Furthermore, aggregation/overriding rules were not clearly documented and could lead to unexpected behaviors (id override).

Proposed resolution

Re-think the aggregation system.

Remaining tasks

Rethink and document.

User interface changes

The aggregation menu will be changed/simplified.

API changes

None.

Data model changes

The way (final) aggregated objects will be modified.

🐛 Bug report
Status

Fixed

Version

1.0

Component

Code

Created by

🇫🇷France guignonv

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @guignonv
  • 🇫🇷France guignonv

    New proposition:

    Only support 2 aggregation modes:

    1. each data source adds properties to the entity of the previous source(s),
    either because the entity data from both sources share the same identifier or
    the new data source entity identifier correspond to a specified field value of
    (one of) the previous source(s) entity (ie. "chaining").

    2. each data source brings its own set of entities. Therefore, entity
    identifiers should all differ from one data source to another. If it is not
    the case, data source entity identifiers could be prefixed by xnttmulti
    using a "virtual group prefix" on a data source-basis.

    For the first case, this module assumes the first data source holds all the
    entity identifiers and can be used for pagging management. If an identifier is
    not present in the first data source but in one (or more) of the next data
    sources, the corresponding entity will be discarded. For filters, each
    data source will be provdied the filters and should ignore irrelevant filters.
    After xnttmulti has aggregated entity data, it will perform an additional
    filtering to apply user filtering on aggregated data as well.

    For the second case, only data sources sharing the same group prefix will be
    aggregated together like in the first case. The first data source of each group
    will be considered as the group "reference" and processed just like in the first
    case for aggregation and pagging management. Entities from data sources without
    group prefix will only be considered and aggregated if their identifiers match
    an earlier group prefix and an existing entity in the corresponding group
    reference. There is a special case when the first(s) data source(s) is (are) not
    in a group. Then, xnttmulti will use the first case behavior and only use the
    next specified group as filters on each data source that has a group.

    By default, xnttmulti will only consider entities from a group data source if
    they actually have the prefix in their identifier. Since some data source may
    not use a prefix in their entity identifier, it is also possible to add as a
    "virtual" prefix with xnttmulti. In that case, any identifier from a given data
    source will be considered and prefixed by each group prefix. It means that if
    more than one virtual prefix is specified for the first time for a given source,
    its data will be virtually duplicated for each prefix (ie. 3 virtual prefixes,
    3 times the same data but with 3 different prefixed identifiers).

    When an entity is aggregated from several data sources (same identifier or with
    chaining), it is possible to choose the way to merge the last data source:
    either without override and only new fields are added, or with override and any
    field from the last source will replace previous existing values, except for
    entity identifier field or chaingin field.

    Let's consider the following examples:

    * Example 1:
    - Data source "alpha":
    - Entity 1: id='A001', field1='abc'
    - Entity 2: id='A002', field1='def', field4='B002'
    - Entity 3: id='X003', field1='ghi'

    - Data source "beta":
    - Entity 1: id='B001', field1='jkl', field2='def'
    - Entity 2: id='B002', field1='mno', field2='ghi', field3='B001', field4='X003'

    - Data source "gamma":
    - Entity 1: id='A001', field1='pqr', field2='def'
    - Entity 2: id='B002', field1='stu', field2='fed'
    - Entity 3: id='X003', field1='vwx', field2='xyz'

    Examples of settings and the corresponding sets of aggregated (final) data:

    - Aggregate all, no group, no override:
    - Entity 1: id='A001', field1='abc', field2='def'
    - Entity 2: id='A002', field1='def', field4='B002'
    - Entity 3: id='X003', field1='ghi', field2='xyz'

    - Aggregate all, no group, with override:
    - Entity 1: id='A001', field1='pqr', field2='def'
    - Entity 2: id='A002', field1='def', field4='B002'
    - Entity 3: id='X003', field1='vwx', field2='xyz'

    - Aggregate all, no group, with override, with gamma identifier as 'field4':
    - Entity 1: id='A001', field1='abc'
    - Entity 2: id='A002', field1='stu', field2='fed', field4='B002'
    - Entity 3: id='X003', field1='ghi'

    - Aggregate by group prefix alpha='A', beta='B', gamma=none, no override:
    - Entity 1: id='A001', field1='abc', field2='def'
    - Entity 2: id='A002', field1='def', field4='B002'
    - Entity 3: id='B001', field1='jkl', field2='def'
    - Entity 4: id='B002', field1='mno', field2='ghi', field3='B001', field4='X003'

    - Aggregate by group prefix alpha='A,X', beta='B', gamma=none, no override:
    - Entity 1: id='A001', field1='abc', field2='def'
    - Entity 2: id='A002', field1='def', field4='B002'
    - Entity 3: id='X003', field1='ghi', field2='xyz'
    - Entity 4: id='B001', field1='jkl', field2='def'
    - Entity 5: id='B002', field1='mno', field2='ghi', field3='B001', field4='X003'

    - Aggregate by group prefix alpha='A', beta='B', gamma='B,X', with override:
    - Entity 1: id='A001', field1='abc'
    - Entity 2: id='A002', field1='def', field4='B002'
    - Entity 3: id='X003', field1='vwx', field2='xyz'
    - Entity 4: id='B001', field1='jkl', field2='def'
    - Entity 5: id='B002', field1='stu', field2='fed', field3='B001', field4='X003'

    * Example 2:
    - Data source "alpha":
    - Entity 1: id='001', field1='abc'
    - Entity 2: id='002', field1='def', field4='002'
    - Entity 3: id='003', field1='ghi'

    - Data source "beta":
    - Entity 1: id='001', field1='jkl', field2='def'
    - Entity 2: id='002', field1='mno', field2='ghi', field3='001', field4='003'

    - Data source "gamma":
    - Entity 1: id='001', field1='pqr', field2='def'
    - Entity 2: id='002', field1='stu', field2='fed'
    - Entity 3: id='003', field1='vwx', field2='xyz'

    Examples of settings and the corresponding sets of aggregated (final) data:

    - Aggregate by group prefix alpha='A', beta='B', gamma=none, no override, with virtual prefix:
    - Entity 1: id='A001', field1='abc'
    - Entity 2: id='A002', field1='def', field4='002'
    - Entity 3: id='A003', field1='ghi'
    - Entity 4: id='B001', field1='jkl', field2='def'
    - Entity 5: id='B002', field1='mno', field2='ghi', field3='001', field4='003'

    - Aggregate by group prefix alpha='A', beta='B', gamma='A,B', with override, with virtual prefix:
    - Entity 1: id='A001', field1='pqr', field2='def'
    - Entity 2: id='A002', field1='stu', field2='fed', field4='002'
    - Entity 3: id='A003', field1='vwx', field2='xyz'
    - Entity 4: id='B001', field1='pqr', field2='def'
    - Entity 5: id='B002', field1='stu', field2='fed', field3='001', field4='003'

    - Aggregate by group prefix alpha=none, beta='002', gamma=none, with override, with gamma identifier as 'field4' (no virtual identifier):
    - Entity 1: id='001', field1='abc'
    - Entity 2: id='002', field1='vwx', field2='xyz', field3='001', field4='003'
    - Entity 3: id='003', field1='ghi'

  • Status changed to Fixed over 1 year ago
  • 🇫🇷France guignonv

    It should be fixed by commit a34ffc34.

  • Automatically closed - issue fixed for 2 weeks with no activity.

Production build 0.71.5 2024