Add an iterate process plugin

Created on 4 May 2022, about 2 years ago
Updated 14 April 2024, 2 months ago

Problem/Motivation

It is currently not possible to iterate over a simple array of values. The 'sub_process' plugin only handles nested arrays where you can define a separate process pipeline for each sub-key. It is not currently possible to simply define a process pipeline and have that execute for each value in a list of values.

Note that despite the similar title this is not related to #2889717: Iterate through rows β†’ as far as I can tell.

Steps to reproduce

-

Proposed resolution

Add an 'iterate' process plugin.

Remaining tasks

  • Agree that this is useful
  • Write tests

User interface changes

-

API changes

A new 'iterate' process plugin.

Data model changes

-

✨ Feature request
Status

Needs review

Version

6.0

Component

Plugins

Created by

πŸ‡©πŸ‡ͺGermany tstoeckler Essen, Germany

Live updates comments and jobs are added and updated live.
  • Needs tests

    The change is currently missing an automated test that fails when run with the original code, and succeeds when the bug has been fixed.

Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • πŸ‡¨πŸ‡­Switzerland Lukas von Blarer

    This works like a charm! Thank you!

    I think that for the specific use case of having an array with values this is the simpler and better solution than the approach of ✨ Add wrapper process plugin to wrap/unwrap values in arrays Needs review .

  • Status changed to Needs review 6 months ago
  • πŸ‡¨πŸ‡­Switzerland Lukas von Blarer

    Sorry, probably that needs a better review ;)

  • πŸ‡ΊπŸ‡ΈUnited States benjifisher Boston area

    I think you can get the same effect using the build_array process plugin from ✨ Process plugin: build an array from source, destination, pipeline Needs work .

    Instead of this example from the doc block for iterate

    process:
      field_dates:
        plugin: iterate
        source: field_timestamps
        process:
          plugin: format_date
          from_format: 'U'
          to_format: 'Y-m-d\TH:i:s'
    

    you can use build_array to prepare the array of arrays that sub_process expects:

    process:
      field_dates:
        - plugin: build_array
          source: field_timestamps
          template:
            - 'pipeline:'
        - plugin: sub_process
          process:
            plugin: format_date
            from_format: 'U'
            to_format: 'Y-m-d\TH:i:s'
            source: '0'
    

    As @danflanagan8 pointed out in #3, it is much simpler in this case to let the Migrate API loop through the array for you, but I think this example shows that any time you want to use iterate, you can get the same effect with build_array and sub_process.

    Many simple uses of sub_process can also be handled by build_array. The second example from #3 becomes

    field_node_references:
      -
        plugin: explode
        delimiter: '/'
        source: field_links
      -
        plugin: build_array
        template:
          - pipeline:2
      -
        plugin: migration_lookup
        migration: node
    

    The build_array plugin uses array_walk_recursive(), which is a lot simpler than sub_process and iterate, both of which create a separate process pipeline. So I prefer to use build_array when it can replace one of the others.

Production build 0.69.0 2024