Add new Tamper that allows multiple columns to be combined into one field

Created on 1 August 2024, 8 months ago

Problem/Motivation

I'm working to import data from spreadsheet that derives from a survey that has "Check all that apply" and each possible response has its own column with a 1 indicated it was checked (or you may commonly see X as well). See below:

[header 1] | q1 = "On which social media platform(s) are you bullied most often?"
----------------------------------------------------------
[header 2] | q1_1 [TikTok] | q1_2 [Instagram] | q1_3 [Twitter/X]
row 1 | x | | x
row 2 | | x | x

etc.

There's not currently a way to add this into a single field that I'm aware. A second Tamper like Explode can be used for a multi-value field in your Drupal entity.

Proposed resolution

I have created a Tamper that gives you this ability.

Remaining tasks

Contribute and add tests.

πŸ“Œ Task
Status

Active

Version

1.0

Component

Code

Created by

πŸ‡ΊπŸ‡ΈUnited States andileco

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @andileco
  • Merge request !27Update file ManyToOneWithReplacement.php β†’ (Open) created by andileco
  • Pipeline finished with Canceled
    8 months ago
    Total: 405s
    #240893
  • Pipeline finished with Success
    8 months ago
    Total: 689s
    #240898
  • Pipeline finished with Success
    8 months ago
    Total: 899s
    #240935
  • Status changed to Needs review 8 months ago
  • πŸ‡ΊπŸ‡ΈUnited States andileco

    Would appreciate any feedback on this.

  • Pipeline finished with Success
    8 months ago
    Total: 917s
    #240977
  • Pipeline finished with Success
    8 months ago
    Total: 684s
    #240996
  • Status changed to Needs work 6 months ago
  • πŸ‡³πŸ‡±Netherlands megachriz

    Thanks for your contribution. I'm trying to understand what this plugin does. I think that this isn't immediately clear. At first glance, it does look like that this plugin only solves problem for a specific use case, but I could be wrong on that.
    I could only figure out what this plugin did, by setting column values to 'X'.

    Given the following config:

    "columns_to_combine" => array:4
      0 => "foo|Foo\r"
      1 => "bar|Bar\r"
      2 => "qux|Qux\r"
      3 => "lorem|Lorem ipsum"
    ]
    "value_marker" => "X"
    "string_separator" => ","
    

    And the following CSV source:

    guid,foo,bar,qux,lorem
    1,X,,,
    2,X,X,X,X
    3,X,,X,X
    

    I get:

    So this plugin sums up which columns are checked and replaces the machine column name with a self defined label.

    I appreciate your contribution, but I'm not sure if we should add this one to Tamper. At least not in its current form. Perhaps it can be made more clear to the user how this plugin should be used:

    • By providing example source input and an example of what result the plugin produces.
    • Tamper plugins have access to the available defined sources, using $this->sourceDefinition->getList() (see the "Rewrite" plugin). It could at least list them, but other possibilities are:
      • Provide a text field for each defined source. This does cause that sources not used in mapping cannot be used, so is less flexible.
      • Provide a select list of all sources, with an option for a custom one. Users can add multiple labels via an "Add" button. Probably is a lot of work to implement though.
      • When putting in the configuration, a sample model is generated from it to immediately reflect what the config does. The model generator would look at what columns are configured and what character is set and based on that it would create a table like the one above.
      • The plugin could also only validate the used sources and warn the user if there are sources used that are not on the list. With an option to ignore the warning. Maybe it would be good enough if the user checks a box "Use undefined column names". Not sure yet about how to name that exactly, I can imagine that "Use undefined column names" is also not really self-explanatory.

  • πŸ‡ΊπŸ‡ΈUnited States andileco

    Hi @megachriz, thanks for your thoughtful review. You see how it works, but are not understanding the value, so let me try to explain:

    Let's say you need to import data from an external survey. Let's pretend have a checkboxes style question: "How did you hear about us? (check as many as apply)":

    The base key for this field is "referral", but there's no referral column with all the selections. Instead, each value option has it's own column, like "referral__a":

    When you import the data into your entity type, you're wanting it to go into a single, unlimited cardinality entityreference field:

    In this case, you would add a second tamper to explode the value generated by this tamper.

    Does that explain a little better the use-case for this?

  • Pipeline finished with Success
    5 months ago
    #331164
  • πŸ‡ΊπŸ‡ΈUnited States spacemuon

    Have you tried the Target - temporary target (temporary_target) option?

    It provides a many to one combined field data solution.

    Required Modules

    Feeds Import, Feeds Tamper and perhaps the Tamper.

    Steps to Combine Field Data

    Step 1 - Ensure the CSV data is in the correct format and contains at least the following columns, e.g. guid, foo, bar, qux, lorem

    Feed Mapping

    Step 2 - Create a Feeds Import with the following Source and Target fields

    Source|Target
    guid | field_id
    title | Name (name)
    foo | temporary target
    bar | temporary target
    qux| temporary target
    lorem | field_combined

    Feed Tamper

    3 - In Tamper, add a Rewrite plugin to rewrite/combine the temporary targets with lorem.

    [foo],[bar],[qux],[lorem]

    4 - Add an Explode plugin to explode the values via a comma or pipe. This option depends on use case, field type, etc.

    Note: Before you try your first import into the field_combined, create a test_field in your content type or vocabulary/taxonomy and map it as the fourth Target field (not a temporary target). This helps ensure the temporary targets and rewrite plugin provide the desired result.

Production build 0.71.5 2024