Optimize migration of specific source IDs for SQL sources

Created on 8 August 2016, over 8 years ago
Updated 28 September 2022, about 2 years ago

Problem/Motivation

Any time a migration is run, the code loops over every row in the source. There is no way to migrate an individual row, or a few selected rows.

There are some source plugins that cannot support anything better, but any source plugin that extends SqlBase should be able to access individual rows efficiently. There may be other sources that can also do this.

Steps to reproduce

Using drush and the Migrate Tools module, try drush mim my_migration --idlist=123. The migration loops over all rows. If there are enough rows, then this can take a long time.

Proposed resolution

Provide an API that some source plugins can implement, allowing specific rows to be migrated.

Remaining tasks

User interface changes

None

API changes

Add an API for selective migrations.

Data model changes

None

Release notes snippet

Original report

The D7 migrate system has a great feature whereby you can import specific IDs, and not have to loop over all IDS in the source query. This is vital for speed when testing a migration and you need to test and entity that has a certain field or odd values. SQLBase should support that by looking for an 'idlist' and altering the source query accordingly. The first user of this feature will be the --idlist option of drush migrate-import (see migrate_plus module).

Feature request
Status

Needs work

Version

11.0 🔥

Component
Migration 

Last updated 1 day ago

Created by

🇺🇸United States moshe weitzman Boston, MA

Live updates comments and jobs are added and updated live.
  • Performance

    It affects performance. It is often combined with the Needs profiling tag.

  • Needs tests

    The change is currently missing an automated test that fails when run with the original code, and succeeds when the bug has been fixed.

  • Needs documentation

    A documentation change is requested elsewhere. For Drupal core (and possibly other projects), once the change has been committed, this status should be recorded in a change record node.

Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • 🇸🇮Slovenia g1smo

    #53 works great for me on drupal 10.2.3.
    Thank you very much for fixing this issue (migrating one piece specific piece of content for debugging purposes used to take 16+ min, now it's done in a couple of seconds on a project I am working on right now).
    Looking forward to seeing this merged into core.

Production build 0.71.5 2024