Add wrapper process plugin to wrap/unwrap values in arrays

Created on 10 October 2022, over 1 year ago
Updated 14 April 2024, 2 months ago

Problem/Motivation

Sometimes I wish a process plugin that is set to "handle_multiples" didn't handle multiples.

Proposed resolution

I ran into another issue that proposes an "iterate" process plugin. ✨ Add an iterate process plugin Needs review . While reviewing that issue, it seemed like a lot of complexity that I'd rather defer to the sub_process plugin, which already handles this kind of thing. The problem is that sub_process requires an array of arrays as input.

A simple way around this would be to add a "wrapper" process plugin that could turn an array into an array of arrays. Then the process plugin I wished didn't handle multiples could be used inside a sub_process. Boom! Oh, and then the "wrapper" process plugin could "unwrap" after the sub_process if needed.

Keeping the "hard part" in sub_process instead of introducing a new "iterate" process plugin makes this very easy to unit test. It's also possible that wrapping or unwrapping a value could be useful outside the context of iteration/sub_processing.

Example:

 * For an input array of arrays, flatten each of child arrays.
 *
 * Assume the input array my_array is:
 * [
 *   ['a', 'b', ['c, 'd']],
 *   [1, 2, [3, 4]],
 * ]
 *
 * And we want to transform this into:
 * [
 *   ['a', 'b', 'c, 'd'],
 *   [1, 2, 3, 4],
 * ]
 *
 * Since the flatten plugin is designed to "handle_multiples",
 * doing this:
 *
 * @code
 * flatten_wont_work:
 *   plugin: flatten
 *   source: my_array
 * @endcode
 *
 * Will result in a single array:
 * ['a', 'b', 'c', 'd', 1, 2, 3, 4]
 *
 * This is where the wrapper plugin can save the day. We can
 * call wrap on my_array, then use a sub_process, and finally
 * unwrap the result of the sub_process.
 *
 * @code
 * my_desired_output:
 *   -
 *     plugin: wrapper
 *     method: wrap
 *     source: my_array
 *     key: element
 *   -
 *     plugin: sub_process
 *     process:
 *       '0':
 *         plugin: flatten
 *         source: element # This should match the 'key' used when wrapping.
 *   -
 *     plugin: wrapper
 *     method: unwrap
 * @endcode
 *
 * In general terms, the most powerful use case is when you wish
 * that an existing process plugin didn't "handle_multiples". In such
 * a case the pattern above can be used: wrap, sub_process, unwrap.

Remaining tasks

N/A

User interface changes

N/A

API changes

New "wrapper" process plugin

Data model changes

N/A

✨ Feature request
Status

Needs review

Version

6.0

Component

Plugins

Created by

πŸ‡ΊπŸ‡ΈUnited States danflanagan8 St. Louis, US

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • πŸ‡ΊπŸ‡ΈUnited States benjifisher Boston area

    I think that wrapper with method: wrap can be replaced by the more flexible build_array plugin from ✨ Process plugin: build an array from source, destination, pipeline Needs work ; and method: unwrap can be replaced by callback with callable: array_pop.

    For example, the pipeline from the issue description,

      -
        plugin: wrapper
        method: wrap
        source: my_array
        key: element
      -
        plugin: sub_process
        process:
          '0':
            plugin: flatten
            source: element # This should match the 'key' used when wrapping.
      -
        plugin: wrapper
        method: unwrap
    

    is equivalent to

      - 
        plugin: build_array
        template:
          element: 'pipeline:'
        source: my_array
      -
        plugin: sub_process
        process:
          '0':
            plugin: flatten
            source: element
      - plugin: callback
        callable: array_pop
    

    I verified that they give the same results on the example input using migrate_sandbox.

    So I suggest we close this issue in favor of ✨ Process plugin: build an array from source, destination, pipeline Needs work .

Production build 0.69.0 2024