Change migration include plugins to use single YAML file per plugin

Created on 30 October 2023, about 1 year ago
Updated 29 November 2023, about 1 year ago

Problem/Motivation

The new include migration plugins are a great feature. However, in practice, the single MYMODULE.migrate_shared_configuration.yml gets big and unwieldy really quickly. It's hard to get an overview of what's there, it's hard to see where one include section ends and another begins, and with multiple developers working on migrations, everyone is working on the same file with resulting git conflicts.

Steps to reproduce

Proposed resolution

Change the include plugins to use one YAML file per include plugin, in a specific subfolder, the same way migration plugins work.

This can be done by changing the plugin manager to use YamlDirectoryDiscovery.

For BC, create a class which decorates the old discovery.

Remaining tasks

User interface changes

API changes

Data model changes

✨ Feature request
Status

Fixed

Version

6.0

Component

Code

Created by

πŸ‡¬πŸ‡§United Kingdom joachim

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @joachim
  • Open in Jenkins β†’ Open on Drupal.org β†’
    Core: 10.1.x + Environment: PHP 8.1 & MySQL 8
    last update about 1 year ago
    36 pass
  • @joachim opened merge request.
  • Status changed to Needs review about 1 year ago
  • πŸ‡¬πŸ‡§United Kingdom joachim
  • πŸ‡ΊπŸ‡ΈUnited States amaisano Boston

    Does this only break it up within the current module, or can it be made so that we can reference different module's shared migration file(s)? Would be nice to have cross-module migration shared yml files (like a family of sub modules where the parent module has a common set of migrations, and the sub modules that each deal with different parts of the migration have more specific shared migration).

  • πŸ‡¬πŸ‡§United Kingdom joachim

    > Does this only break it up within the current module, or can it be made so that we can reference different module's shared migration file

    The migration includes are merely plugins -- you can reference any plugin of that type in any migration. That's already the case with the current code.

  • πŸ‡ΊπŸ‡ΈUnited States amaisano Boston

    So cross-module use of the shared_configuration YAML data is possible w/o this patch? Like so:

    /my-migrations
    --/modules
    --/--/node_migration
    --/--/--/node_migration.migrate_shared_configuration.yml (with a "defaults" include definition)
    --/--/term_migration
    --/--/--/term_migration.migrate_shared_configuration.yml (with a "taxonomy" include definition)
    --/--/misc_migration
    --/--/--/migrations
    --/--/--/--/misc_migration.yaml (with contents below)
    
    # Misc Migration
    id: misc_migration
    includes:
      - defaults
      - taxonomy
    source: ...

    ?

  • πŸ‡¬πŸ‡§United Kingdom joachim

    AFAICT, yes. I had a quick look in the code of MigrateIncludeHandler and it's not doing any doctoring to the 'include' value.

  • πŸ‡ΊπŸ‡ΈUnited States amaisano Boston

    It would be nice if you could still group multiple inside a single YML file, especially when you have multiple override "snippets" that are only a few lines -- creating a separate file for all of them is a bit overwhelming.

    defaults.yml
    - include1:
    -   source:
    -     constants:
    -       a: 1
    - include2:
    -   source:
    -     constants:
    -       b: 2
    - include3:
    -   source:
    -     constants:
    -       c: 3
    
    big_complex_field.yml
    - process:
    -   field_paragraphs:
    -      plugin: ...   
  • πŸ‡ΊπŸ‡ΈUnited States amaisano Boston

    Also, sub-folders under migrate_shared_configuration/ would be nice:

    - migrate_shared_configuration/node/bundle_1.yml
    - migrate_shared_configuration/node/bundle_2.yml
    - migrate_shared_configuration/term/vocab_1.yml
  • Status changed to RTBC about 1 year ago
  • Status changed to Fixed about 1 year ago
  • Automatically closed - issue fixed for 2 weeks with no activity.

Production build 0.71.5 2024