Contrib.social

Using "/" item_selector in Json parser plugin config does not work as expected anymore

Open on Drupal.org →
Created on 4 August 2024, 4 months ago

Problem/Motivation

The 6.0.3 release introduced a change in the JSON parser plugin API which made my migrations to report zero rows. With Migrate Plus "> 6.0.3", using a / item selector was valid. But now, both in 6.0.3, 6.0.4 (and also in dev) it is evaluated as an invalid item selector.
This is source plugin configuration:

  plugin: url
  data_fetcher_plugin: file
  item_selector: "/"
  data_parser_plugin: json
  urls: [...]
  ids: [...]
  fields: [...]

In 6.0.2 and before, in the parser plugin, the logic in regards of the item selector was this:

// Otherwise, we're using xpath-like selectors.
$selectors = explode('/', trim((string) $this->itemSelector, '/'));
foreach ($selectors as $selector) {
  if (is_array($source_data) && array_key_exists($selector, $source_data)) {
    $source_data = $source_data[$selector];
  }
}
return $source_data;

Right now, it is a bit different, and this is the change which makes my migration to fail:

// If the item_selector is an empty string, return all.
if ($item_selector === '') {
  return $this->sourceData;
}

// Otherwise, we're using xpath-like selectors.
$selectors = explode('/', trim($item_selector, '/'));
$return = $this->sourceData;
foreach ($selectors as $selector) {
  // If the item_selector is missing, return an empty array.
  if (!isset($return[$selector])) {
    return [];
  }
  $return = $return[$selector];
}
return $return;

Proposed resolution

  1. Restore BC in regards of the previously accepted (and processed) item_selector configuration...
  2. ...but, log a warning about they are deprecated from 6.0.3 (or 6.0.5 ?)

Remaining tasks

  1. Decide whether using only slash is acceptable or not. (The MR I will create accepts it.)
  2. Decide whether the support of broken / invalid selectors is really deprecated or not. (I assume these selectors are deprecated.)

User interface changes

Nothing.

API changes

BC restored as in 6.0.2; but broken selectors are deprecated.

Data model changes

🐛 Bug report
Status

Active

Version

6.0

Component

Plugins

Created by

🇭🇺Hungary huzooka Hungary 🇭🇺🇪🇺

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @huzooka
  • Merge request !102Issue #3465782: Using "/" item_selector in Json parser plugin config does not work as expected anymore → (Open) created by huzooka
  • Pipeline finished with Failed
    4 months ago
    Total: 247s
    #243472
  • Merge request !103Resolve #3465782 Full backward compatibility for JSON parser plugin's item_selector → (Open) created by huzooka
  • Pipeline finished with Failed
    4 months ago
    Total: 280s
    #243531
  • Pipeline finished with Failed
    4 months ago
    Total: 248s
    #243534
  • Pipeline finished with Failed
    4 months ago
    Total: 237s
    #243539
  • Pipeline finished with Failed
    4 months ago
    Total: 263s
    #244053
  • Issue was unassigned.
  • Status changed to Needs review 4 months ago
  • Comment 4 months ago →
    🇭🇺Hungary huzooka Hungary 🇭🇺🇪🇺

    Created two MRs:
    * 3465782-using-slash-json-parser (MR 102) restores support for "/" item selector.
    * 3465782-full-bc-json-parser (MR 103) fully restores the previous behavior with (faulty) item selectors. (Look at the new test in JsonTest).

    I don't think I can do anything for the "max PHP version" tests - seems that the phpunit.xml is build for a Drupal version which still had "HtmlOutputPrinter", but the actual artifact contains Drupal 11, which does not have the class anymore.

    Asking for review (and further ideas).

  • Comment 3 months ago →
    🇨🇭Switzerland steva1982

    Hi,
    I have the same problem because my migration YML file had item_selector: /.
    Looking for other issues, I found only this morning this trick 📌 Add note about item_selector: 0 in Migrate JSON Example module RTBC and it works for me.
    Probably is not the best solution, but for the moment it allows me to import.

  • Comment 3 months ago →
    🇦🇺Australia sonnykt Melbourne, Australia

    Confirm the issue when using the JSON data_parser with item_selector: /. getResponseContent() returns the correct data but getSourceData() always returns an empty array.

    My debugging also leads to the same line which was changed in this MR https://git.drupalcode.org/project/migrate_plus/-/merge_requests/81/diff....

    As a workaround, I have to use item_selector: '' to make migrations work again.

  • Comment 3 months ago →
    🇺🇸United States dasginganinja Bethlehem, PA

    We just realized today that this stopped our migrations from running.
    We had the following value: item_selector: /events/event

    After updating this to `events` this started working in 6.0.4.

    Our item selectors were all using the format of `event/field_name` already. I'm surprised this was working in the first place. 😜

  • Comment 5 days ago →
    🇺🇸United States alison

    Our JSON source migrations are also still broken, on 6.0.4, and fixed/fine when we revert to 6.0.2.

    I wasn't able to apply MR 102 or 103 to 6.0.4, I haven't tried with 6.0.x yet, I was pausing first, because GitLab says the pipelines failed with those anyway (but I might still try).

    (I'm afraid I don't have other ideas to contribute!)

  • Comment 2 days ago →
    🇩🇪Germany szeidler Berlin

    For some of our migrations it also has been a breaking change. item_selector: '/' we have been able to change to item_selector: 0, which kept the desired elements selected (in our case a JSON that directly starts with an array of objects).

contrib.social Blog FAQ Discussions
Production build 0.71.5 2024