Contrib.social

Using "/" item_selector in Json parser plugin config does not work as expected anymore

Open on Drupal.org →
Created on 4 August 2024, 6 months ago

Problem/Motivation

The 6.0.3 release introduced a change in the JSON parser plugin API which made my migrations to report zero rows. With Migrate Plus "> 6.0.3", using a / item selector was valid. But now, both in 6.0.3, 6.0.4 (and also in dev) it is evaluated as an invalid item selector.
This is source plugin configuration:

  plugin: url
  data_fetcher_plugin: file
  item_selector: "/"
  data_parser_plugin: json
  urls: [...]
  ids: [...]
  fields: [...]

In 6.0.2 and before, in the parser plugin, the logic in regards of the item selector was this:

// Otherwise, we're using xpath-like selectors.
$selectors = explode('/', trim((string) $this->itemSelector, '/'));
foreach ($selectors as $selector) {
  if (is_array($source_data) && array_key_exists($selector, $source_data)) {
    $source_data = $source_data[$selector];
  }
}
return $source_data;

Right now, it is a bit different, and this is the change which makes my migration to fail:

// If the item_selector is an empty string, return all.
if ($item_selector === '') {
  return $this->sourceData;
}

// Otherwise, we're using xpath-like selectors.
$selectors = explode('/', trim($item_selector, '/'));
$return = $this->sourceData;
foreach ($selectors as $selector) {
  // If the item_selector is missing, return an empty array.
  if (!isset($return[$selector])) {
    return [];
  }
  $return = $return[$selector];
}
return $return;

Proposed resolution

  1. Restore BC in regards of the previously accepted (and processed) item_selector configuration...
  2. ...but, log a warning about they are deprecated from 6.0.3 (or 6.0.5 ?)

Remaining tasks

  1. Decide whether using only slash is acceptable or not. (The MR I will create accepts it.)
  2. Decide whether the support of broken / invalid selectors is really deprecated or not. (I assume these selectors are deprecated.)

User interface changes

Nothing.

API changes

BC restored as in 6.0.2; but broken selectors are deprecated.

Data model changes

🐛 Bug report
Status

Active

Version

6.0

Component

Plugins

Created by

🇭🇺Hungary huzooka Hungary 🇭🇺🇪🇺

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @huzooka
  • Merge request !102Issue #3465782: Using "/" item_selector in Json parser plugin config does not work as expected anymore → (Open) created by huzooka
  • Pipeline finished with Failed
    6 months ago
    Total: 247s
    #243472
  • Merge request !103Resolve #3465782 Full backward compatibility for JSON parser plugin's item_selector → (Open) created by huzooka
  • Pipeline finished with Failed
    6 months ago
    Total: 280s
    #243531
  • Pipeline finished with Failed
    6 months ago
    Total: 248s
    #243534
  • Pipeline finished with Failed
    6 months ago
    Total: 237s
    #243539
  • Pipeline finished with Failed
    6 months ago
    Total: 263s
    #244053
  • Issue was unassigned.
  • Status changed to Needs review 6 months ago
  • Comment 6 months ago →
    🇭🇺Hungary huzooka Hungary 🇭🇺🇪🇺

    Created two MRs:
    * 3465782-using-slash-json-parser (MR 102) restores support for "/" item selector.
    * 3465782-full-bc-json-parser (MR 103) fully restores the previous behavior with (faulty) item selectors. (Look at the new test in JsonTest).

    I don't think I can do anything for the "max PHP version" tests - seems that the phpunit.xml is build for a Drupal version which still had "HtmlOutputPrinter", but the actual artifact contains Drupal 11, which does not have the class anymore.

    Asking for review (and further ideas).

  • Comment 5 months ago →
    🇨🇭Switzerland steva1982

    Hi,
    I have the same problem because my migration YML file had item_selector: /.
    Looking for other issues, I found only this morning this trick 📌 Add note about item_selector: 0 in Migrate JSON Example module RTBC and it works for me.
    Probably is not the best solution, but for the moment it allows me to import.

  • Comment 5 months ago →
    🇦🇺Australia sonnykt Melbourne, Australia

    Confirm the issue when using the JSON data_parser with item_selector: /. getResponseContent() returns the correct data but getSourceData() always returns an empty array.

    My debugging also leads to the same line which was changed in this MR https://git.drupalcode.org/project/migrate_plus/-/merge_requests/81/diff....

    As a workaround, I have to use item_selector: '' to make migrations work again.

  • Comment 5 months ago →
    🇺🇸United States dasginganinja Bethlehem, PA

    We just realized today that this stopped our migrations from running.
    We had the following value: item_selector: /events/event

    After updating this to `events` this started working in 6.0.4.

    Our item selectors were all using the format of `event/field_name` already. I'm surprised this was working in the first place. 😜

  • Comment 2 months ago →
    🇺🇸United States alison

    Our JSON source migrations are also still broken, on 6.0.4, and fixed/fine when we revert to 6.0.2.

    I wasn't able to apply MR 102 or 103 to 6.0.4, I haven't tried with 6.0.x yet, I was pausing first, because GitLab says the pipelines failed with those anyway (but I might still try).

    (I'm afraid I don't have other ideas to contribute!)

  • Comment about 2 months ago →
    🇩🇪Germany szeidler Berlin

    For some of our migrations it also has been a breaking change. item_selector: '/' we have been able to change to item_selector: 0, which kept the desired elements selected (in our case a JSON that directly starts with an array of objects).

  • Comment 11 days ago →
    🇺🇸United States stacypendell

    Our feeds started importing successfully with MR 102 in version 6.0.5. Did not test MR 103 because we only need the slash. No ideas to suggest, but planning to try some item selector variations as others have suggested in the comments here.

contrib.social Blog FAQ Discussions
Production build 0.71.5 2024