Contrib.social

Run migrations in parallel

Open on Drupal.org →
Created on 18 February 2020, about 5 years ago
Updated 14 March 2025, about 1 month ago

A few months ago, on my project with 120 or so migrations (some with several hundred thousand rows to migrate), I was looking at speeding up the throughput by running migrations in parallel, respecting dependencies. I wrote a drush command (submitted at ✨ Generate graphs of migration dependencies Active ) to graph dependencies to help people work out what can run in parallel with what. Shortly after that, it occurred to me that we don't need people to figure that out. Thus was born:

$ drush help mimp
Run all migrations with maximum parallelism.

Examples:
  migrate:import-parallel --max-processes=5 --status-frequency=5 Runs up to 5 migrations in parallel as soon as their dependencies are fulfilled, with status of running migrations
                                                                 output every 5 minutes.

Options:
  --max-processes[=MAX-PROCESSES]       Maximum number of processes to run at once.
  --status-frequency[=STATUS-FREQUENCY] Number of minutes between status updates.

Aliases: mimp

It's been working pretty well for us, so I'm (belatedly) posting it here.

Ideally, I think rather than a separate command it should be an optional feature of drush migrate-import - if --max-processes is present and greater than one, spawn off the subprocesses, passing on any other options to the subprocesses (adding --skip-progress-bar, because it'll be really messy otherwise).

I don't have time now to write such a patch (maybe April? - anyone who wants to tackle it before then is welcome to), but in the meantime some people might find this useful.

✨ Feature request
Status

Active

Version

6.0

Component

Drush commands

Created by

🇺🇸United States mikeryan Murphysboro, IL, USA

Live updates comments and jobs are added and updated live.
  • Needs tests

    The change is currently missing an automated test that fails when run with the original code, and succeeds when the bug has been fixed.

Sign in to follow issues

Merge Requests

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • Merge request !22WIP: Issue #3114358: Run migrations in parallel → (Open) created by Anybody
  • First commit to issue fork.
  • Pipeline finished with Failed
    over 1 year ago
    #19288
  • Open in Jenkins → Open on Drupal.org →
    Core: 10.1.x + Environment: PHP 8.1 & MySQL 8
    last update over 1 year ago
    PHPLint Failed
  • Pipeline finished with Failed
    over 1 year ago
    #19290
  • Pipeline finished with Failed
    over 1 year ago
    #19303
  • Comment over 1 year ago →
    🇷🇺Russia qzmenko Novosibirsk

    qzmenko → changed the visibility of the branch 6.0.x to hidden.

  • Comment over 1 year ago →
    🇷🇺Russia qzmenko Novosibirsk

    qzmenko → changed the visibility of the branch 6.0.x to active.

  • Comment about 1 month ago →
    🇩🇪Germany Grevil

    grevil → changed the visibility of the branch 6.0.x to hidden.

  • Comment about 1 month ago →
    🇩🇪Germany Grevil

    Rebased the issue forks issue branch.

    Here is a static patch from the rebased MR.

  • Pipeline finished with Failed
    about 1 month ago
    Total: 257s
    #448196
  • Comment about 1 month ago →
    🇩🇪Germany Grevil

    Sorry, that patch was empty...

  • Pipeline finished with Failed
    about 1 month ago
    Total: 205s
    #448203
  • Comment about 1 month ago →
    🇩🇪Germany Grevil

    grevil → changed the visibility of the branch 6.0.x to active.

  • Comment about 1 month ago →
    🇩🇪Germany Grevil

    Ok, the current implementation doesn't work anymore. "migrationsList()" is already implemented, and once removed, the "_construct()" method throws the next error, expecting less parameters, and the list probably goes on after this, seeing the changes made by @imash in 6.0.x. But those changes seem too much? Unsure about this. Either way also using the updated 6.0.x by @imash and adjusting the namespace also throws an error when migrating:

    [error] Error: Call to a member function getMigrationDependencies() on array in Drupal\migrate_tools\Drush\Commands\MigrateToolsCommands->importParallel() (line 1022 of /var/www/html/web/modules/contrib/migrate_tools/src/Drush/Commands/MigrateToolsCommands.php) #0 [internal function]: Drupal\migrate_tools\Drush\Commands\MigrateToolsCommands->importParallel()

    And generally I don't think this is the correct approach for parallel migration, as we implement a new standalone "migrate:import-parallel" command, instead of extending the existing "migrate:import" command and adding a "--parallel" flag. I understand that this isn't easily implemented, but in the current implementation we drop crucial flags like "--continue-on-failure" or "--group=...", which at least for us, is a no-go.

contrib.social Blog FAQ Discussions
Production build 0.71.5 2024