Provide ability to reference current value of process pipeline as a source property

Created on 20 September 2021, over 3 years ago
Updated 18 November 2023, about 1 year ago

Problem/Motivation

Several core migrate process plugins support a list of source properties. These include:

  • concat
  • callback
  • download
  • file_copy
  • menu_link_parent
  • null_coalesce
  • route
  • static map

Currently, these plugins cannot be used with a list of source plugins unless they are the first process plugin in a pipeline. For example let's say you had products and you wanted to process the name so that it has a lowercase i in front of a capitalized name. For example, "product" will become "iProduct". Currently we need a temporary destination property to deal with this.

source:
  plugin: embedded_data
  data_rows:
    -
      'id': 1
      'field_product': 'product'
  ids:
    id:
      type: string
  constants:
    i: 'i'
process:
  temp_value:
    plugin: callable
    source: field_product
    callable: ucfirst
  processed_value:
    plugin: concat
    source:
      - 'constants/i'
      - '@temp_value'

Proposed resolution

Create a keyword for referencing the current value of the process pipeline. I propose %pipeline. The process section above could then be rewritten as:

process:
  processed_value:
    -
      plugin: callable
      source: field_product
      callable: ucfirst
    -    
       plugin: concat
       source:
         - 'constants/i'
         - '%pipeline'

Either way the result for processed_value would be iProduct.

Remaining tasks

TBD

User interface changes

TBD

API changes

TBD

Data model changes

TBD

Release notes snippet

TBD

✨ Feature request
Status

Closed: works as designed

Version

9.3

Component
MigrationΒ  β†’

Last updated about 12 hours ago

Created by

πŸ‡ΊπŸ‡ΈUnited States danflanagan8 St. Louis, US

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • πŸ‡ΊπŸ‡ΈUnited States danflanagan8 St. Louis, US

    My mind was just blown. I was thinking about this again, this time thinking that I'd have a SuperGet class that would override Get. Then I saw that Get can already be coaxed into referencing the current value of the pipeline. From its transform method:

    foreach ($properties as $property) {
          if ($property || (string) $property === '0') {
            $return[] = $row->get($property);
          }
          else {
            $return[] = $value;
          }
        }
    

    That "else" case puts the current pipeline value into $return. When does that case run? When $property is null an empty string or an empty array. Those are essentially keywords that do the thing I was introducing with %pipeline So we already can do this! The snippet from the IS would be:

    process:
      processed_value:
        -
          plugin: callable
          source: field_product
          callable: ucfirst
        -    
           plugin: concat
           source:
             - 'constants/i'
             - null
    

    It's weird and not intuitive and hard to read, but it totally works!!!

Production build 0.71.5 2024