Parsing JSON - arrays

Created on 12 October 2019, about 5 years ago
Updated 4 March 2024, 10 months ago

EDIT: added the yml and fixed the json data (was a copy/paste error)

Problem: Not able to get the data (id or Name) from the "Sets" array. Goal is to create 4 contents out of this data.

This is the source:

{
  "Time" : "2019-10-08",
  "Centers" : [ {
    "id" : "B01502",
    "CenterId" : 1502,
    "Sets" : [ {
      "id" : "B0150200",
      "Name" : "name_valuex"
    }, {
      "id" : "B0150201",
      "Name" : "name_valueu"
    } ]
  }, {
    "id" : "B01503",
    "CenterId" : 1503,
    "Sets" : [ {
      "id" : "B0150301",
      "Name" : "name_valueb"
    }, {
      "id" : "B0150302",
      "Name" : "name_valuea"
    } ]
  } ]
}

This is my YML

id: test_data
source:
  constants:
    TITLE_PREFIX: 'title prefix - '

  plugin: url
  data_fetcher_plugin: file
  data_parser_plugin: json

  urls:
    - /test.json
    
  item_selector: 1
  fields:
    - name: id
      label: 'id'
      selector: id
  ids:
    id:
      type: string
process:
  title: constants/TITLE_PREFIX
  field_presetid: id 
destination:
  plugin: 'entity:node'
  default_bundle: bundle_name

I have tried to put different values to "item_selector:" like /Centers/Sets or "0" etc.
Modules:
Migrate: Version: 8.7.8
Migrate Plus: Version: 8.x-4.2
Migrate Tools: Version: 8.x-4.4
Drupal: 8.7.8
PHP 7.2.22

💬 Support request
Status

Active

Version

6.0

Component

Miscellaneous

Created by

🇫🇮Finland jukka792

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • Merge request !87#3239180: Start a JSON subitems data parser → (Merged) created by marvil07
  • Pipeline finished with Success
    10 months ago
    Total: 246s
    #111060
  • Issue was unassigned.
  • Status changed to Needs review 10 months ago
  • 🇵🇪Peru marvil07

    It will be nice to have a way to walk through a set of items nested in all the item selector items.

    I have started one custom data parser plugin that does that.
    It basically extends the json data parser plugin, and then uses a given selector to unify its values across all the elements.

    E.g. the following the original example, the following source configuration bits will extract the four items.

    data_parser_plugin: json_subitem
    item_selector: Centers
    subitem_selector: Sets
    

    On the ideal world, jq pipelines can do the work once, but if the source endpoints cannot be changed, something like this data parser helps to produce a set that is sensible to walk.

    New commits on new 3087614-json-subitems topic branch and related new MR !87.

    - 3f81967 #3239180: Start a JSON subitems data parser
    
  • Status changed to RTBC 4 months ago
  • 🇺🇦Ukraine init90 Ukraine

    I'm using the code above and it works well.

  • First commit to issue fork.
  • 🇺🇸United States slbrassard

    The parser was not working when using a pager because getSourceData always assumed we wanted the subitems. I added a check for the item selector parameter being the data item selector before getting subitems. I also removed the getSourceData function return type since it could be anything as with the parent function.

  • Pipeline finished with Success
    about 2 months ago
    Total: 325s
    #324216
  • Pipeline finished with Skipped
    29 days ago
    #345051
  • heddn Nicaragua

    Thxs for the contributions

  • Automatically closed - issue fixed for 2 weeks with no activity.

Production build 0.71.5 2024