Migrate paragraphs into node with entity_generate

Created on 21 September 2021, almost 3 years ago
Updated 19 January 2024, 5 months ago

Hello,

I'm working on D9. I have a contentType named 'trip' which has a field of type paragraph (a list) with a paragraph type of type "travel_participant" (it's a simple name + surname paragraph).
I need to import the data from an existing DB, but the data is split in several tables (for simplicity's sake I work with CSVs)

The import comes from two separate CSV: the first one imports the `trip` into the correct Drupal Nodes and proper Content Type. This import works as expected.
The `trip` is imported and its field, name `trip_participants` is empty (as expected).

The second CSV is like this

"id","trip_id","name","surname","pos" 
"199751","33201","Emanuele","Degennaro" 
"199752","33201","Francesca","Stendardo" 
"199804","33202","Chiara","Gnocchi" 
"199641","33203","Valentina","Bizzini"  

What I'd like to achieve is to update the existing `trip` Node adding on the fly one or more paragraphs created during the migration. I know I could write a custom plugin, but I was wondering if the `entity_generate` process plugin could work and be simpler.
.
I've seen many examples that do a two step import, first the paragraphs and then the reference id in the node. I don't really see the point in doing this (the paragraphs only exists because of the trip) and moreover I am not sure as to how to link the created paragraphs with the final node (maybe add a fake field holding the original trip_id, and then use mappings, like in this link about understanding migrations)

At the moment my YAML looks like this (some lines omitted for brevity)

id: trip_participants_csv_import
# ... omitted lines
source:
  plugin: csv
  path: /Users/walter/trip_participants.csv 
  ids:
    - id 
  fields:
    -
      name: id
    -
      name: trip_id
    -
      name: name
    -
      name: surname 
process:
  nid:
    plugin: migration_lookup
    migration: trip_csv_import
    source: trip_id
  field_participants: #this is the field holding a list of paragraphs of type 'travel_participant`
    -
      plugin: entity_generate
      entity_type: paragraph
      source: surname  # this is useless, but it's needed b Entyty_generate to create paragraph instance, since it wants a value
      value_key: id # ? what's it for?  the plugin complains if it 's not present 
      bundle: travel_participant
      bundle_key: type
      values:
        field_surname: surname
        field_name: name
destination:
  plugin: 'entity:node'
  default_bundle: trip
migration_dependencies:
  required:
    - trip_csv_import

One paragraph is created (i can see it in the Drupal DB table `paragraphs_item` and also in the `paragraph_field_surname/name` tables), with the correct values, but the migration fails complaining that it cannot insert in the field 'field_participants" a simple ID (in this case the id of the newly created paragraph). I am learning by debugging the plugins but it's very difficult and long, so maybe I just got the wrong syntax, or maybe it's something impossible right now.

BTW I see with the debugger that the entity_lookup works as expected: it retrieves the original node via mapping of the old `trip_id` to the new `nid` in Drupal

The error happens in `EntityReferenceRevisionsItem.php` in `setValue` where it tries to set the field to an entity, but it gets a single ID. The code expects an object or an array with `target_id` or `target_revision_id`, but all it gets is a number.

I think I am not that far from getting what I want, but i don't know how to tell the migration plugin `entity_generate` to pass along the whole object instead of just the ID.

Any idea?
Thanks
W

πŸ’¬ Support request
Status

Active

Version

5.1

Component

Plugins

Created by

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • πŸ‡¬πŸ‡§United Kingdom very_random_man

    In case anyone comes across this like I did, looking for a way to to use entity_generate with entity_reference_revisions, here's how you can do it just with some process magic. In the example below I have a phone field from a D7 field source (field_contact_phone) and I'm using that to generate a custom multi_field entity which has separate label and phone fields. The important part is the last bit which looks up the revision ID using an entity_value plugin.

    
      field_phone:
        - plugin: sub_process
          source: field_contact_phone
          process:
            _linklabel: linklabel
            target_id:
              plugin: entity_generate
              source: phonenumber
              entity_type: multi_field
              bundle: labelled_phone
              value_key: field_phone_number
              bundle_key: bundle
              values:
                label: '@_linklabel'
                field_phone_number/0/value: phonenumber
            _revision_id_value:
              plugin: entity_value
              source: '@target_id'
              entity_type: multi_field
              field_name: revision_id
            target_revision_id: '@_revision_id_value/0/value'
    
    
  • πŸ‡¬πŸ‡§United Kingdom natts London

    Thanks for that @very_random_man. I was doing something more simple with a entity reference field for media, and didn't know that I needed to use target_id as the attribute of the field to assign the generated entity to it, but your code showed me that's what I was missing:

    process:
      youtube_1/url: youtube1
      youtube_2/url: youtube2
      youtube_3/url: youtube3
      vimeo_1/url: vimeo
    
      videos:
        - plugin: get
          source:
            - '@youtube_1'
            - '@youtube_2'
            - '@youtube_3'
            - '@vimeo_1'
    
      field_videos:
        plugin: sub_process
        source: '@videos'
        process:
          target_id:
            - plugin: skip_on_empty
              source: url
              method: process
            - plugin: entity_generate
              source: url
              entity_type: media
              bundle: remote_video
              bundle_key: bundle
              value_key: field_media_oembed_video
              values:
                field_media_oembed_video: url
  • πŸ‡©πŸ‡ͺGermany vistree

    @very_random_man - you saved my day!! I were able to migrate paragraphs with your method without any problems ;-)
    Thanx so much!!

    My paragraph (colorscheme) has 5 input fields:

    • id (unique identifiere)
    • hex
    • type
    • color (name)
    • menu (to be used within the menu)
      field_colorscheme:
        - plugin: skip_on_empty
          method: process
          source: colorscheme
        -
          plugin: sub_process
          process:
            _menu: menu
            _hex: hex
            _type: type
            _color: color
            target_id:
              plugin: entity_generate
              source: id
              entity_type: paragraph
              bundle: color_set
              value_key: field_remote_id
              bundle_key: type
              values:
                field_remote_id/0/value: id
                field_color_menu/0/value: '@_menu'
                field_color_value/0/color:  '@_hex'
                field_title/0/value:  '@_type'
                field_color_type/0/value:  '@_color'
            _revision_id_value:
              plugin: entity_value
              source: '@target_id'
              entity_type: paragraph
              field_name: revision_id
            target_revision_id: '@_revision_id_value/0/value'
    
  • πŸ‡¬πŸ‡§United Kingdom nicholasThompson

    Another handy way of using entity_generate, granted not paragraphs here, but might help someone (or Future-Me)

      _imageLookup:
        plugin: migration_lookup
        migration: migrate_file_managed
        source: field_image/0/fid
    
      field_media_image:
        plugin: entity_generate
        access_check: false
        entity_type: media
        source: '@_imageLookup'
        value_key: name
        ignore_case: true
        default_values:
          bundle: 'image'
        values:
          name: '@_imageLookup/filename'
          field_media_image: '@_imageLookup'

    This creates an entity:image during a node migrate.

  • πŸ‡©πŸ‡ͺGermany vistree

    Anybody an idea on how to use entity_generate to add NON existing files to media items?
    I am in trouble with source and destination ...

  • πŸ‡¬πŸ‡§United Kingdom joekers UK

    The entity_value plugin here was the key for me - thanks @very_random_man!

  • πŸ‡ΊπŸ‡¦Ukraine Bohdan Vasyliuk

    I have successfully generated paragraphs using entity_generate. However, when I executed migrate:rollback, the nodes were removed, while the paragraphs generated by the migration remained in the database.
    Consequently, there is a risk of having numerous orphan paragraphs, which may be related to this issue: https://www.drupal.org/project/paragraphs/issues/3056819 πŸ› Remove orphan paragraphs when host entity is removed Closed: duplicate . Please correct me if I'm wrong.

  • πŸ‡¬πŸ‡§United Kingdom Baysaa

    @Bohdan that's correct, things created using entity_generate won't have a migrate_map table created for them, so they won't get rolled back naturally.

    You'll have to snapshot/create restore points for your DB while doing these things I found.

Production build 0.69.0 2024