Execute each migration in a separate cron queue

Created on 16 May 2018, over 6 years ago
Updated 25 September 2024, 4 months ago

If there are more number of migrations all of them might not be able to complete import during a single cron run. It would be good if all of them could be separately queued and imported.

Good article covering queue api at - https://www.sitepoint.com/drupal-8-queue-api-powerful-manual-and-cron-qu...

📌 Task
Status

Active

Version

1.0

Component

Code

Created by

🇮🇳India ajits India

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • 🇬🇧United Kingdom 2dareis2do

    I have tested migrate_queue_importer. One interesting thing about migrate_queue_importer is it seems to use hook cron to elevate the priority of the module as hook_cron tends to be:

    1. executed before the queue jobs are run.
    2. executed sequentially, i.e. one hook implementation followed by another.

    One issue is that it still treats migrations as a batch. e.g. say I have a rss feed that has 100 items on it which may in turn have one or more dependencies with a similar amount of items, the chances of a job completing in say 30 seconds are slim which is the default value for withsuspendMaximumWait.

    Now if a migration was executed and imported row by row, using the queue might be more beneficial. It is more likely a single row could be imported within this 30 second period or similar, rather than abstracting the import using migrate tools UI or drush (much of the functionality from migrate tools has been replicated in drush) where it processes the queue as a single job. In fact this is different to migrate Ui that does use the batch api to process items one by one, This allows a job to continue from if it was interrupted.

    Of course migrate also has its own state (db) and ways of knowing how to handle an import depending on if it has been imported successfully or not etc.

    I think it might be beneficial to have a way of importing a migration row by row, while at the same time allowing users to override the suspendMaximumWait to suit, so that a migration import and any dependencies works more optimally with the queue API and is likely to time out or fail.

Production build 0.71.5 2024