Prevent running the same migration twice

Created on 19 January 2023, almost 2 years ago

Problem/Motivation

There's a possible edge case with long-running migrations when multiple Drush cron runners could be running at the same time (i.e. if migration takes more time than a default cron execution lock timeout).

In this case, the second drush process may start the same migration (and reset its status, mark all items for update etc). This could lead to application slowdown, overwriting the same data multiple times, possible conflicts with entities created with migration lookup etc.

Steps to reproduce

1. Create a large migration with lots of data that takes e.g. 1 hour to complete.
2. Enable running this migration with Cron.
3. Run `drush cron`.
4. Wait for 15 minutes (this is a `cron` lock expiration time, ref `\Drupal\Console\Command\Cron\ExecuteCommand::execute`); otherwise, Drupal won't let you run another cron process..
5. Run `drush cron` again.

Expected: The second process does not start the migration since it's already running. A second migration may be executed though.
Actual: The second process resets the migration status and starts it over again.

Proposed resolution

The cron job should acquire a lock while starting the migration, then subscribe to Migrate row process event and renew the lock periodically (to make sure it's not expired by timeout).
Release the lock after migration ends.
The cron job should not start migrations if the lock could not be acquired.

Remaining tasks

Do it.

User interface changes

None.

API changes

None.

Data model changes

Nope.

✨ Feature request
Status

Active

Version

1.0

Component

Code

Created by

πŸ‡ΊπŸ‡¦Ukraine abramm Lutsk

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

No activities found.

Production build 0.71.5 2024