SQL source plugins: allow defining conditions and join in migration yml

Created on 23 July 2019, almost 5 years ago
Updated 24 January 2024, 5 months ago

Problem/Motivation

For custon migration it's a very common request to filter the migrated items. Some real use cases are:
- migrate nodes older than {date};
- skip blocked users or users with certain emails or users that never accessed the website;
- migrate users with certain roles;

It would be great to be able to use conditions to narrow down your SQL source data for content/config entity types. The resulting functionality would provide much more capabilities to how you can migrate nodes one content type at a time, and the same for taxonomy terms -- but for users (by role), menu links (by menu), and so on.

The functionality would let you narrow your data during the source/extraction phase, instead of during the process/transformation phase (which you can do using skip_on_value/empty + method: row). Benefits would include general tidiness of data :) And! Speed/performance!

Proposed resolution

Introduce new configuration properties in SqlBase base class: conditions, joins and distinct:
conditions - to filter the results;
joins - to join additional tables (required to filter by fields/properties stored in separate tables);
distinct - to avoid duplicated results when joining one-to-many records;

If such configuration is present, it's being applied to the query in prepareQuery method, when the base query is already defined.

Remaining tasks

  1. Review and validate the approach;
  2. Resolve πŸ› SqlBase::prepareQuery() should be called also on count Needs work , which is kind of a blocker (the source count is not updated after applying conditions, because the count() ignoring prepareQuery method);
  3. Add more tests;
  4. Write a change record;
✨ Feature request
Status

Needs work

Version

11.0 πŸ”₯

Component
MigrationΒ  β†’

Last updated about 2 hours ago

Created by

πŸ‡ΊπŸ‡ΈUnited States alison

Live updates comments and jobs are added and updated live.
  • Needs tests

    The change is currently missing an automated test that fails when run with the original code, and succeeds when the bug has been fixed.

  • Needs change record

    A change record needs to be drafted before an issue is committed. Note: Change records used to be called change notifications.

Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • πŸ‡ΊπŸ‡ΈUnited States alison

    I know this is both old and has up-in-the-air elements to it, BUT, I successfully applied MR #849 to a Drupal 10.1 site, and then successfully migrated users of just one role. (I only applied the changes in MR #849, I didn't grab anything from #2833060 πŸ› SqlBase::prepareQuery() should be called also on count Needs work .)

    I did get warnings when I ran drush mim, but it still ran successfully -- but I'll share them anyway:

     [warning] Undefined array key "conditions" SqlBase.php:365
     [warning] foreach() argument must be of type array|object, null given SqlBase.php:365
     [warning] Undefined array key "joins" SqlBase.php:370
     [warning] foreach() argument must be of type array|object, null given SqlBase.php:370
     [warning] Undefined array key "fields" SqlBase.php:375
     [warning] foreach() argument must be of type array|object, null given SqlBase.php:375
     [warning] Undefined array key "conditions" SqlBase.php:365
     [warning] foreach() argument must be of type array|object, null given SqlBase.php:365
     [warning] Undefined array key "joins" SqlBase.php:370
     [warning] foreach() argument must be of type array|object, null given SqlBase.php:370
     [warning] Undefined array key "fields" SqlBase.php:375
     [warning] foreach() argument must be of type array|object, null given SqlBase.php:375
    

    My very rough take on those warnings is that they mean my conditions syntax could/should be better. (I copied from one of the code examples in SqlBase.php, but maybe I certainly may have misinterpreted it.)

    Speaking of which, here's the "source" section of my user migration YML:

    source:
      plugin: d7_user
      joins:
        -
          table: users_roles
          alias: ur
          condition: u.uid = ur.uid
      conditions:
        -
          field: ur.rid
          value: [6]
          operator: IN
      distinct: TRUE
    
Production build 0.69.0 2024