Allow callback for Url source, and single item Json plugin

Reading through the Drupalspoons issue, with the state we now have on d.o. perhaps we should reopen the issue here and create MR from here? The feature req. it still useful and from what I can see, we still do not have a good alternative?

Manual reroll and minor changes just to get the ball rolling. My local tests are still running so partly untested.

@Steinmb Agreed. This patch needs to be updated in module. As there are scenario to keep the urls fetch from backend configuration. the custom callback will be the solution to achieve it

Am I correct that the list of URLs created for a combination of list/item pattern and pagination would be something like this?

• https://source.data.com/api/items?start=1&end=200
• https://source.data.com/api/items/1
• https://source.data.com/api/items/2
• https://source.data.com/api/items/3
• https://source.data.com/api/items/...
• https://source.data.com/api/items?start=201&end=400
• https://source.data.com/api/items/201
• https://source.data.com/api/items/202
• https://source.data.com/api/items/203
• ...

If the URL callback just does the job of listing all these endpoints, does anyone see a potential source data handling conflict by treating the list and item responses the same? Seems the separate solutions offered by #2608610: Add support for list/item pattern → and 📌 Support paging through multiple requests Needs work might solve for that, but I am open to getting a better grasp of this solution.

Thoughts?

Actually, come to think of it, the list/item solution I went with was a variation of the UrlPerRow example in Drupal Answer link from #26 of #2608610: Add support for list/item pattern → .

We wrote a similar plugin extension that provides a list of URLs back to the main source plugin. There hasn't been any conflict with # of items found vs. imported vs. any other status - it just works like normal.

We've extended the the migrate_plug\data_parse\JSON plugin, and altered the nextSource() method to build this list.

We are crawling a JSON:API in our custom plugin until we find no more "next-page" attributes in the JSON response, and then compile the full list of URLs as basically a bunch of offset page requests.

I can share the code if anyone is interested.

@amaisano id very much like to read the code of this custom JSON plugin! :)

@amaisano me too! please share the code of this custom JSON plugin!

I tried this patch (42, 3040427-42-migrate_plus_multiple_urls.patch → ) and code for function my_module_migrate_urls but I have this error:

In Url.php line 88:
The URL callback function is not callable.

What I am missing to make callback function callable?
Thanks in advance

Here is the parser code plugin we made that handles custom/dynamic/multiple source urls:

https://www.drupal.org/project/migration_jsonapi →

We are using patch #42 for years now very successfully with many different endpoints. Moving to reviewed & tested.

Migrate sources are so vastly different that - for me - it is much easier and faster to write a PHP callback than to find and study and update 437 different list/item source plugin modules. And very often those plugins are NOT (re-) usable because endpoint A returns an array [] on root, endpoint B returns an object [] on root, endpoint C returns an HTTP-Code 404 when the last page number was passed and endpoint D just returns an HTTP200 with no content. PHP callbacks just work. For years. (Sorry for the rant, but the more different sources you have to migrate from, the more useful this simple patch is.)

+1 from me to include this feature into migrate_plus.

We are also using this patch to great success. I am curious if it's possible to add an argument that can be passed to the callback as well. Our use case is that we have several migrations that all use very similar code, but have different endpoints to get different resources. So, I envision something like this:

source:
  plugin: url
  data_fetcher_plugin: http
  data_parser_plugin: json
  urls:
    callback: my_custom_callback
    argument: something-passed-here

And then in the callback function:

function my_custom_callback(MigrationInterface $migration, $argument) {

}

This may be another issue/MR, but wanted to see if anyone else had this kind of use case. At the moment, the workaround is for us to use multiple callback functions, one for each migration.

I just did a test and I can do something like this and it works - not sure how valid it is though:

Migration yml:

source:
  plugin: url
  data_fetcher_plugin: http
  data_parser_plugin: json
  urls:
    callback: my_custom_callback
  callback_arg: 'my amazing argument'

Callback function:

function my_custom_callback(MigrationInterface $migration) {
  $my_arg = $migration->getPluginDefinition()['source']['callback_arg'];
}

1. +++ b/src/Plugin/migrate/source/Url.php
   @@ -32,11 +71,23 @@ class Url extends SourcePluginExtension {
   +        $configuration['urls'] = $configuration['urls']['callback']($migration);
   

   Rather than load up a potentially HUGE list of URLs into the configuration, we should defer this to getSourceUrls(), and the fetchers should call that.

   Also, changing configuration feels a bit ick -- it should remain what it was set to in the plugin definition rather than being spookily changed.