Migrate in D9

Created on 4 October 2018, over 6 years ago

Updated 27 June 2023, almost 2 years ago

Problem/Motivation

I think the biggest problem people are going to have with this is that people are waiting for Drupal 9 to open to do these big changes, but the point that d9 opens it’s actually too late. Big breaking changes need to be built and finished in D8 in the next 12-18 months before d8 goes LTS.

We need to plan and figure out what pieces we want to see in D9 migrate.

Problem/Motivation

Some random thoughts compiled in a recent migrate maintainers meeting with phenaproxima, mikelutz, maxocub and masipila in attendance. More could probably be added. The general idea is that with Drupal 9 coming, what can we do to make the migration code more maintainable. These notes take a brief look at each of the plugin types.

There was some interest to review work from mikeryan on an independent migrate system he was working on. The only look that the maintainers have seen was at DC Nashville, so we'd like to see if there is any benefit from using it under the hood.

Plugins:

Fields - probably don't need a lot of change. We'll still need them for D7=>D9 upgrades. Hopefully not for D8=D9 though.
Source - Use generators instead of just iteratable. See if we can use FilterIterator for filtering out rows a.k.a. SkipRowException, Have a stronger/tighter schema for getIds. Never allow conditionals inside of a getIds as to what columns are allowed.
- #3017237: Use generators by default in source plugins →
- 📌 Alter derived migration definition early in life Needs work
Destination - no major changes probably needed, same logic for source plugins apply about getIds. No conditional logic based on if multilingual capabilities are enabled or not, etc. These can be added later by enabling a module and then the number of columns in the map get broken.
idmap - should use named keys for source and destinations; not just sorted index arrays, be less tied to Sql and use the db abstraction layer a little more
process - should be able to use any callable, should be a plugin collection instead of an array, is there a means to easily introduce yml for conditional routes in a process pipeline.
- #3252562: In Callback Migrate process, document how to use functions that accept no argument as callable →
- #2882276: Extended callback process plugin to call functions with multiple parameters →
migration - should replace the executable, remove all vestiges of config
- 📌 Move memory management from MigrateExecutable to an event subscriber Needs review

Other random thoughts,

signaling using exceptions should be killed dead. Use flags and other signals to control skipping a.k.a. SkipRowException, SkipProcessException
Convert plugins to use PluginCollections: #3004510: Migrations should use plugin collections for source, destination, and ID map →
Add a metadata concept to migrations so we can store non-configuration metadata related to a migration. Or maybe it is configuration??? See #3

Priorities

Process plugins, process plugins, process pipeline.

Convert to using plugin collections
- #3004510: Migrations should use plugin collections for source, destination, and ID map →
Lookup bc breaking plugin to replace migrate_lookup
- #3246666: Deprecate migrate_lookup and replace with cleaner process plugin →

idmap
migration plugin takes over executable

🌱 Plan

Status

Closed: outdated

Version

11.0 🔥

Component

Migration →

Last updated 3 days ago

Maintained by
🇺🇸United States @benjifisher
@heddn
🇺🇸United States @mikelutz
🇳🇿New Zealand @quietone

Created by

heddn Nicaragua

Live updates comments and jobs are added and updated live.

Drupal 9

Incomplete comments

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

Comment almost 2 years ago →
🇳🇿New Zealand quietone
We managed to fix a couple of issues in the Issue Summary!

We are well into Drupal 10 now so I am closing this in favor of #3280243: [Meta] Migrate in D10 → .

contrib.social Blog FAQ Discussions

Production build 0.71.5 2024