Contrib.social

Create a "pipeline" process plugin to re-use YAML config

Open on Drupal.org →
Created on 30 March 2020, about 4 years ago
Updated 19 February 2024, 4 months ago

Problem/Motivation

We often need to copy/paste YAML config across several migrations.

If we want to apply a specific suite of process plugins to all fulltext fields (e.g: dom process plugins), all you can do ATM is copy/paste the suite of yaml config everywhere.
Same if you have a specific process plugin with specific/complex config.

Proposed resolution

Define a "pipeline" process plugin that defines a suite of process plugins to run.

Given the following example in a migration:

  process:
  'body/value':
    - plugin: pipeline
      id: my_process_pipeline
      source: 'body/0/value'

We would create the file migrate_plus.migration_process_pipeline.my_process_pipeline.yml with config:

langcode: en
status: true
id: my_process_pipeline
label: 'My process pipeline'
description: 'Process pipeline for processing fulltext fields'
process:
  -
    plugin: dom
    method: import
  -
    plugin: dom_str_replace
    mode: attribute
    xpath: '//a'
    attribute_options:
      name: href
    search: 'foo'
    replace: 'bar'
  -
    plugin: dom_str_replace
    mode: attribute
    xpath: '//a'
    attribute_options:
      name: href
    regex: true
    search: '/foo/'
    replace: 'bar'
  -
    plugin: dom
    method: export
✨ Feature request
Status

Needs work

Version

5.0

Component

Plugins

Created by

🇧🇪Belgium herved

Live updates comments and jobs are added and updated live.
  • Needs tests

    The change is currently missing an automated test that fails when run with the original code, and succeeds when the bug has been fixed.

  • Needs issue summary update

    Issue summaries save everyone time if they are kept up-to-date. See Update issue summary task instructions.

Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • Comment 10 months ago →
    🇺🇸United States amaisano Boston

    This doesn't work with sub_process plugins. The snippet is sending the 1st item in the array rather than the whole array to the separate file.

  • Comment 10 months ago →
    🇺🇸United States danflanagan8 St. Louis, US

    Would it work to put the single_value plugin in the pipeline ahead of the snippet plugin in that case?

    testing:
  -
    plugin: single_value
    source: my_array
  -
    plugin: snippet
    module: my_module
    path: my_process_pipeline
  • Comment 10 months ago →
    🇺🇸United States amaisano Boston

    @danflanagan8 that seems to be working! What a nice workaround. Thank you.

    Snippet YML:

    -
  plugin: sub_process
  source: bricks
  process:
    _stub:
      plugin: switch_on_condition
      source: type
      .....

    --migrate-debug output:

    ...
    13 => []
    14 => array:3 [
      "_stub" => array:2 [
        0 => "227"
        1 => "274"
      ]
      "target_id" => "227"
      "target_revision_id" => "274"
    ]

    It's interesting how the output values contain the pseudo_field, which usually isn't present in the output for Destinations. It has no effect on the import thankfully.

  • Comment 4 months ago →
    🇺🇸United States alison

    I just started using #19 to do identical processing on a bunch of rich text fields and I'm in love, this feature is absolutely wonderful, THANK YOU!!

    The status is "Needs work" not "Needs review," so I won't change it to RTBC, but do y'all know what's left needing work, or maybe can we move this to RTBC........?

  • Comment 4 months ago →
    🇺🇸United States danflanagan8 St. Louis, US

    The `snippet` plugin for sure needs test coverage. (Added the tag.) And the IS here describes a different approach. I would be in favor of updating the IS to describe the `snippet` plugin. So I added that tag too.

contrib.social Blog FAQ Discussions
Production build 0.69.0 2024