Provide ability to reference current value of process pipeline as a source property

Created on 20 September 2021, over 3 years ago
Updated 18 November 2023, over 1 year ago

Problem/Motivation

Several core migrate process plugins support a list of source properties. These include:

  • concat
  • callback
  • download
  • file_copy
  • menu_link_parent
  • null_coalesce
  • route
  • static map

Currently, these plugins cannot be used with a list of source plugins unless they are the first process plugin in a pipeline. For example let's say you had products and you wanted to process the name so that it has a lowercase i in front of a capitalized name. For example, "product" will become "iProduct". Currently we need a temporary destination property to deal with this.

source:
  plugin: embedded_data
  data_rows:
    -
      'id': 1
      'field_product': 'product'
  ids:
    id:
      type: string
  constants:
    i: 'i'
process:
  temp_value:
    plugin: callable
    source: field_product
    callable: ucfirst
  processed_value:
    plugin: concat
    source:
      - 'constants/i'
      - '@temp_value'

Proposed resolution

Create a keyword for referencing the current value of the process pipeline. I propose %pipeline. The process section above could then be rewritten as:

process:
  processed_value:
    -
      plugin: callable
      source: field_product
      callable: ucfirst
    -    
       plugin: concat
       source:
         - 'constants/i'
         - '%pipeline'

Either way the result for processed_value would be iProduct.

Remaining tasks

TBD

User interface changes

TBD

API changes

TBD

Data model changes

TBD

Release notes snippet

TBD

✨ Feature request
Status

Closed: works as designed

Version

9.3

Component
MigrationΒ  β†’

Last updated 2 days ago

Created by

πŸ‡ΊπŸ‡ΈUnited States danflanagan8 St. Louis, US

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • πŸ‡ΊπŸ‡ΈUnited States danflanagan8 St. Louis, US

    My mind was just blown. I was thinking about this again, this time thinking that I'd have a SuperGet class that would override Get. Then I saw that Get can already be coaxed into referencing the current value of the pipeline. From its transform method:

    foreach ($properties as $property) {
          if ($property || (string) $property === '0') {
            $return[] = $row->get($property);
          }
          else {
            $return[] = $value;
          }
        }
    

    That "else" case puts the current pipeline value into $return. When does that case run? When $property is null an empty string or an empty array. Those are essentially keywords that do the thing I was introducing with %pipeline So we already can do this! The snippet from the IS would be:

    process:
      processed_value:
        -
          plugin: callable
          source: field_product
          callable: ucfirst
        -    
           plugin: concat
           source:
             - 'constants/i'
             - null
    

    It's weird and not intuitive and hard to read, but it totally works!!!

Production build 0.71.5 2024