Add XPath-style filtering ability in JSON data parser plugin

Created on 18 October 2018, about 6 years ago
Updated 14 September 2023, about 1 year ago

In the current iteration of the data parser plugin `Drupal\migrate_plus\Plugin\migrate_plus\data_parser\Json`, the ability to use XPath selectors only allows for simple string selectors at each level. It would be helpful to allow for a slightly more advanced form of XPath-based filtering when parsing JSON data. For example, allowing for filtering based on the child attributes of the current selector level.

Testing instructions

It is a little tricky to test this patch, since an important part is that it adds a new package to composer.json.

One approach is to add the patch as usual, to get everything else (the code that uses the added package). Then

composer require softcreatr/jsonpath:0.8.*

This should have the same effect as letting Composer install the new requirement.

The other approach is to add a repository to composer.json for the issue fork for this issue in composer.json:

"repositories": [
        {
            "type": "vcs",
            "url": "https://git.drupalcode.org/issue/migrate_plus-3007709"
        },
        {
            "type": "composer",
            "url": "https://packages.drupal.org/8"
        }
    ],

The order matters! Then require the feature branch for this issue. (This will also update composer.json.)

composer require drupal/migrate_plus:dev-3007709-xpath-6.0.x

This will pull in the patch for this issue and install the newly added package.

JSONPath Information

The following links are helpful when trying to create proper JSONPath selectors with queries.

✨ Feature request
Status

Needs review

Version

6.0

Component

Plugins

Created by

πŸ‡ΊπŸ‡ΈUnited States michael_wojcik

Live updates comments and jobs are added and updated live.
  • Needs tests

    The change is currently missing an automated test that fails when run with the original code, and succeeds when the bug has been fixed.

Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • Open on Drupal.org β†’
    Core: 9.5.x + Environment: PHP 8.0 & MySQL 5.7
    last update over 1 year ago
    Not currently mergeable.
  • @benjifisher opened merge request.
  • πŸ‡ΊπŸ‡ΈUnited States benjifisher Boston area

    I am updating the version to 6.0.x. I also created a new MR based on that branch and the patch in #40.

  • πŸ‡ΊπŸ‡ΈUnited States benjifisher Boston area
  • πŸ‡ΊπŸ‡ΈUnited States pixlkat

    I installed the MR version of migrate_plus using the instructions above on a vanilla D10 site with the minimal profile and created a test content type to use as the destination for the test migration.

    I created module with a simple migration using a sample JSON file source (attached) and was successful in using a selector to limit the items which were migrated. The JSON contains an attribute named "favoriteFruit"; the item_selector in the migration selects those with favoriteFruit=strawberry. My migration only imported the items with the matching selector with no errors.

    My simple migration:

    id: test_data
    label: Migrate Test Data
    
    source:
      plugin: url
      data_fetcher_plugin: file
      data_parser_plugin: jsonpath
      urls:
        - modules/custom/migration_test/data/people.json
    
      item_selector: '$.people[?(@.favoriteFruit=strawberry)]'
    
      fields:
        -
          name: guid
          label: 'GUID'
          selector: guid
        -
          name: name
          label: 'Person name'
          selector: name
        -
          name: age
          label: 'Age'
          selector: age
        -
          name: gender
          label: 'Gender'
          selector: gender
        -
          name: company
          label: 'Company'
          selector: company
        -
          name: about
          label: 'About'
          selector: about
        -
          name: favorite_fruit
          label: 'Favorite Fruit'
          selector: favoriteFruit
    
      ids:
        guid:
          type: string
    
    process:
      type:
        plugin: default_value
        default_value: people
      title: name
      field_age: age
      field_gender: gender
      field_company: company
      body/value: about
      body/format:
        plugin: default_value
        default_value: plain_text
      field_favorite_fruit: favorite_fruit
    
    destination:
      plugin: 'entity:node'
    
    migration_dependencies: { }
    dependencies:
      enforced:
        module:
          - migration_test
    
  • πŸ‡ΊπŸ‡ΈUnited States pixlkat
  • Open in Jenkins β†’ Open on Drupal.org β†’
    Core: 9.5.x + Environment: PHP 8.1 & MySQL 8
    last update over 1 year ago
    192 pass
  • Status changed to Needs work over 1 year ago
  • πŸ‡ΊπŸ‡ΈUnited States benjifisher Boston area

    @pixlcat:

    Thanks for testing this, and for adding JSONPath links to the issue summary.

    Since we looked at this issue at DrupalCon earlier this month, I am adding the tag for that.

    I reviewed the changes. It mostly looks good, but I would like a little code cleanup. I am setting the status to NW, and I will re-review promptly. Let's try to get this issue done!

    One of my review comments will require a bit of research and/or experimentation.

  • Open in Jenkins β†’ Open on Drupal.org β†’
    Core: 9.5.x + Environment: PHP 8.1 & MySQL 8
    last update over 1 year ago
    192 pass
  • Status changed to Needs review about 1 year ago
  • πŸ‡ΊπŸ‡ΈUnited States pixlkat

    @benjifisher I guess I didn't set this back to needs review. Doing that here.

Production build 0.71.5 2024