Add new Tamper that allows multiple columns to be combined into one field

Created on 1 August 2024, 5 months ago

Problem/Motivation

I'm working to import data from spreadsheet that derives from a survey that has "Check all that apply" and each possible response has its own column with a 1 indicated it was checked (or you may commonly see X as well). See below:

[header 1] | q1 = "On which social media platform(s) are you bullied most often?"
----------------------------------------------------------
[header 2] | q1_1 [TikTok] | q1_2 [Instagram] | q1_3 [Twitter/X]
row 1 | x | | x
row 2 | | x | x

etc.

There's not currently a way to add this into a single field that I'm aware. A second Tamper like Explode can be used for a multi-value field in your Drupal entity.

Proposed resolution

I have created a Tamper that gives you this ability.

Remaining tasks

Contribute and add tests.

πŸ“Œ Task
Status

Active

Version

1.0

Component

Code

Created by

πŸ‡ΊπŸ‡ΈUnited States andileco

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @andileco
  • Merge request !27Update file ManyToOneWithReplacement.php β†’ (Open) created by andileco
  • Pipeline finished with Canceled
    5 months ago
    Total: 405s
    #240893
  • Pipeline finished with Success
    5 months ago
    Total: 689s
    #240898
  • Pipeline finished with Success
    5 months ago
    Total: 899s
    #240935
  • Status changed to Needs review 5 months ago
  • πŸ‡ΊπŸ‡ΈUnited States andileco

    Would appreciate any feedback on this.

  • Pipeline finished with Success
    5 months ago
    Total: 917s
    #240977
  • Pipeline finished with Success
    5 months ago
    Total: 684s
    #240996
  • Status changed to Needs work 2 months ago
  • πŸ‡³πŸ‡±Netherlands megachriz

    Thanks for your contribution. I'm trying to understand what this plugin does. I think that this isn't immediately clear. At first glance, it does look like that this plugin only solves problem for a specific use case, but I could be wrong on that.
    I could only figure out what this plugin did, by setting column values to 'X'.

    Given the following config:

    "columns_to_combine" => array:4
      0 => "foo|Foo\r"
      1 => "bar|Bar\r"
      2 => "qux|Qux\r"
      3 => "lorem|Lorem ipsum"
    ]
    "value_marker" => "X"
    "string_separator" => ","
    

    And the following CSV source:

    guid,foo,bar,qux,lorem
    1,X,,,
    2,X,X,X,X
    3,X,,X,X
    

    I get:

    So this plugin sums up which columns are checked and replaces the machine column name with a self defined label.

    I appreciate your contribution, but I'm not sure if we should add this one to Tamper. At least not in its current form. Perhaps it can be made more clear to the user how this plugin should be used:

    • By providing example source input and an example of what result the plugin produces.
    • Tamper plugins have access to the available defined sources, using $this->sourceDefinition->getList() (see the "Rewrite" plugin). It could at least list them, but other possibilities are:
      • Provide a text field for each defined source. This does cause that sources not used in mapping cannot be used, so is less flexible.
      • Provide a select list of all sources, with an option for a custom one. Users can add multiple labels via an "Add" button. Probably is a lot of work to implement though.
      • When putting in the configuration, a sample model is generated from it to immediately reflect what the config does. The model generator would look at what columns are configured and what character is set and based on that it would create a table like the one above.
      • The plugin could also only validate the used sources and warn the user if there are sources used that are not on the list. With an option to ignore the warning. Maybe it would be good enough if the user checks a box "Use undefined column names". Not sure yet about how to name that exactly, I can imagine that "Use undefined column names" is also not really self-explanatory.

  • πŸ‡ΊπŸ‡ΈUnited States andileco

    Hi @megachriz, thanks for your thoughtful review. You see how it works, but are not understanding the value, so let me try to explain:

    Let's say you need to import data from an external survey. Let's pretend have a checkboxes style question: "How did you hear about us? (check as many as apply)":

    The base key for this field is "referral", but there's no referral column with all the selections. Instead, each value option has it's own column, like "referral__a":

    When you import the data into your entity type, you're wanting it to go into a single, unlimited cardinality entityreference field:

    In this case, you would add a second tamper to explode the value generated by this tamper.

    Does that explain a little better the use-case for this?

  • Pipeline finished with Success
    about 2 months ago
    #331164
Production build 0.71.5 2024