Includes not merging in correct order

Created on 17 August 2023, over 1 year ago

Problem/Motivation

When you provide multiple includes in your migration, it is expected they will overwrite (if the same keys exist) in order of the listed includes in the migration YML (with the actual migration overwriting any includes).

However, in the case below, "translation" is being overwritten by "defaults" even though the "translation" include is both listed last, and is defined last in the shared config YML.

Steps to reproduce

Shared configuration YML:

# Used by all migrations as a base.
defaults:
  status: true
  migration_tags:
    - sw
    - top-down
  source:
    jsonapi_query_params:
      filter:
        default_langcode: 1
    fields:
      -
        name: langcode
        label: 'Language'
        selector: attributes/langcode
  process:
    langcode: langcode

# Base setup for jsonapi parser.
source_parser_jsonapi:
  source:
    data_fetcher_plugin: http
    data_parser_plugin: jsonapi
    item_selector: data
    urls: {  }

# List of languages to use for translation migrations.
source_languages:
  source:
    jsonapi_langcodes:
      - fr
      - es
      - ja
      - de

# Defaults for any translation-based migration.
translation:
  migration_tags:
    - translation
  source:
    jsonapi_query_params:
      filter:
        default_langcode: 0
    ids:
      langcode:
        type: string
  destination:
    translations: true

Translation-based migration YML:

id: migrate_sw_node_wall_translation
include:
  - defaults
  - source_parser_jsonapi
  - source_languages
  - translation
migration_tags:
  - node
  - node_wall
label: 'Node - Wall (Translations)'
source:
  plugin: url
  jsonapi_endpoint: /jsonapi/node/wall
  ...

When running the above migration, it is expected that the extended parser plugin (jsonapi) will use the default_langcode: 0 for its config (used to add "&filter[default_langcode]=0" to the urls).

However, migrate_devel is showing that it is compiled as:

  "jsonapi_query_params" => array:1 [
    "filter" => array:1 [
      "default_langcode" => 1
    ]
  ]

Putting the "translation" include above "defaults" makes "default_langcode" use the correct value from the shared "translation" config:

include:
  - translation # <--- putting this on top makes its values take precedence
  - defaults
  - source_parser_jsonapi
  - source_languages
#  - doesn't work here

Proposed resolution

???

Remaining tasks

???

User interface changes

???

API changes

???

Data model changes

???

πŸ› Bug report
Status

Active

Version

6.0

Component

Code

Created by

πŸ‡ΊπŸ‡ΈUnited States amaisano Boston

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @amaisano
  • πŸ‡ΊπŸ‡ΈUnited States amaisano Boston

    Question: Do custom plugins (like the parser plugin being described here), need to implement something specific to be 100% compatible with the shared configuration system? It could be something custom in that code could be throwing this off? I can share the code if desired.

  • πŸ‡ΊπŸ‡ΈUnited States amaisano Boston
Production build 0.71.5 2024