Handling null array

Hi Ste,
I suspect this is something that would be better handled in your source plugin. What source plugin are you using? What is the error that you are seeing?

Hi @danflanagan8,

My source plugin is a URL:

source:
  # `plugin` inherited from migration group.
  # `data_fetcher_plugin` inherited from migration group.
  # `data_parser_plugin` inherited from migration group.
  # `urls` inherited from migration group.
  urls:
  - https://my-website.com/api/custom-page/retrieve.json?mt=video&nitems=99

The error is:

 [error]  TypeError: Drupal\migrate_plus\Plugin\migrate_plus\data_parser\Json::selectByDepth(): Argument #1 ($raw_data) must be of type array, null given, called in /var/opt/www/usilu_test/docs/web/modules/contrib/migrate_plus/src/Plugin/migrate_plus/data_parser/Json.php on line 72 in Drupal\migrate_plus\Plugin\migrate_plus\data_parser\Json->selectByDepth() (line 103 of /var/opt/www/usilu_test/docs/web/modules/contrib/migrate_plus/src/Plugin/migrate_plus/data_parser/Json.php) #0 /var/opt/www/usilu_test/docs/web/modules/contrib/migrate_plus/src/Plugin/migrate_plus/data_parser/Json.php(72): Drupal\migrate_plus\Plugin\migrate_plus\data_parser\Json->selectByDepth()
#1 /var/opt/www/usilu_test/docs/web/modules/contrib/migrate_plus/src/Plugin/migrate_plus/data_parser/Json.php(130): Drupal\migrate_plus\Plugin\migrate_plus\data_parser\Json->getSourceData()
#2 /var/opt/www/usilu_test/docs/web/modules/contrib/migrate_plus/src/DataParserPluginBase.php(153): Drupal\migrate_plus\Plugin\migrate_plus\data_parser\Json->openSourceUrl()
#3 /var/opt/www/usilu_test/docs/web/modules/contrib/migrate_plus/src/DataParserPluginBase.php(96): Drupal\migrate_plus\DataParserPluginBase->nextSource()
#4 /var/opt/www/usilu_test/docs/web/modules/contrib/migrate_plus/src/DataParserPluginBase.php(87): Drupal\migrate_plus\DataParserPluginBase->next()
#5 [internal function]: Drupal\migrate_plus\DataParserPluginBase->rewind()
#6 /var/opt/www/usilu_test/docs/web/modules/contrib/migrate_plus/src/DataParserPluginBase.php(229): iterator_count()
#7 /var/opt/www/usilu_test/docs/web/core/modules/migrate/src/Plugin/migrate/source/SourcePluginBase.php(539): Drupal\migrate_plus\DataParserPluginBase->count()
#8 /var/opt/www/usilu_test/docs/web/core/modules/migrate/src/Plugin/migrate/source/SourcePluginBase.php(509): Drupal\migrate\Plugin\migrate\source\SourcePluginBase->doCount()
#9 /var/opt/www/usilu_test/docs/web/modules/contrib/migrate_tools/src/EventSubscriber/MigrationDrushCommandProgress.php(69): Drupal\migrate\Plugin\migrate\source\SourcePluginBase->count()
#10 /var/opt/www/usilu_test/docs/web/modules/contrib/migrate_tools/src/Drush/Commands/MigrateToolsCommands.php(1065): Drupal\migrate_tools\EventSubscriber\MigrationDrushCommandProgress->initializeProgress()
#11 /var/opt/www/usilu_test/docs/web/modules/contrib/migrate_tools/src/Drush/Commands/MigrateToolsCommands.php(483): Drupal\migrate_tools\Drush\Commands\MigrateToolsCommands->executeMigration()
#12 [internal function]: Drupal\migrate_tools\Drush\Commands\MigrateToolsCommands->import()
#13 /var/opt/www/usilu_test/docs/vendor/consolidation/annotated-command/src/CommandProcessor.php(276): call_user_func_array()
#14 /var/opt/www/usilu_test/docs/vendor/consolidation/annotated-command/src/CommandProcessor.php(212): Consolidation\AnnotatedCommand\CommandProcessor->runCommandCallback()
#15 /var/opt/www/usilu_test/docs/vendor/consolidation/annotated-command/src/CommandProcessor.php(176): Consolidation\AnnotatedCommand\CommandProcessor->validateRunAndAlter()
#16 /var/opt/www/usilu_test/docs/vendor/consolidation/annotated-command/src/AnnotatedCommand.php(391): Consolidation\AnnotatedCommand\CommandProcessor->process()
#17 /var/opt/www/usilu_test/docs/vendor/symfony/console/Command/Command.php(326): Consolidation\AnnotatedCommand\AnnotatedCommand->execute()
#18 /var/opt/www/usilu_test/docs/vendor/symfony/console/Application.php(1096): Symfony\Component\Console\Command\Command->run()
#19 /var/opt/www/usilu_test/docs/vendor/symfony/console/Application.php(324): Symfony\Component\Console\Application->doRunCommand()
#20 /var/opt/www/usilu_test/docs/vendor/symfony/console/Application.php(175): Symfony\Component\Console\Application->doRun()
#21 /var/opt/www/usilu_test/docs/vendor/drush/drush/src/Runtime/Runtime.php(110): Symfony\Component\Console\Application->run()
#22 /var/opt/www/usilu_test/docs/vendor/drush/drush/src/Runtime/Runtime.php(40): Drush\Runtime\Runtime->doRun()
#23 /var/opt/www/usilu_test/docs/vendor/drush/drush/drush.php(139): Drush\Runtime\Runtime->run()
#24 /var/opt/www/usilu_test/docs/vendor/drush/drush/drush(4): require('...')
#25 /var/opt/www/usilu_test/docs/vendor/bin/drush(119): include('...')
#26 {main}. 

Ok, this is definitely a source plugin thing and can't be fixed using Migrate Conditions.

This might require a slightly modified data_parser plugin. I had a project not long ago where I made a modest extension of the Json data_parser to deal with Json that had a slightly different structure than what the Json parser expects:

/**
 * Obtain JSON data for migration...when the JSON response is super flat.
 *
 * @DataParser(
 *   id = "flat_json",
 *   title = @Translation("Flat JSON")
 * )
 */
class FlatJson extends Json {

  /**
   * The standard json parser expects the response to be an array of arrays.
   * This plugin handles a JSON response that is flat array, allowing the
   * response to be treated as a single row for the migration.
   *
   * {@inheritdoc}
   */
  protected function getSourceData(string $url): array {
    $response = $this->getDataFetcherPlugin()->getResponseContent($url);

    // Convert objects to associative arrays.
    $source_data = json_decode($response, TRUE, 512, JSON_THROW_ON_ERROR);

    // If json_decode() has returned NULL, it might be that the data isn't
    // valid utf8 - see http://php.net/manual/en/function.json-decode.php#86997.
    if (is_null($source_data)) {
      $utf8response = utf8_encode($response);
      $source_data = json_decode($utf8response, TRUE, 512, JSON_THROW_ON_ERROR);
    }

    return [$source_data];
  }

}

You might be able to do something similar where you introduce special handling for NULL data.

I'm going to close this issue since it's outside the realm of Migrate Conditions. The Drupal Slack #migration channel would be a great place to ask for additional help here. Good luck and happy migrating!

ok, I understood.
Thank you for the suggestion.

Automatically closed - issue fixed for 2 weeks with no activity.