Migrations parse the previously uploaded file

Created on 18 November 2024, 3 days ago

Problem/Motivation

This is a follow up to my colleague, caesius' comment in ✨ Allow developers to expose more configuration in the migrate form Fixed :

We have an old migration that is still regularly used to import data on a site. We are unable to update to version 1.1 of Migrate source UI module because of the issue that caesius mentions, which will block us from upgrading to D11 until it's resolved.

It appears that the migration will attempt to parse the previous version of the file prior to the migration. I suspect that is because despite the new migration source being set in $options['configuration'] it still loads the migration object with the original source, and parses it during the $source->rewind(); step.

We aren't really certain why it needs to parse the previous file. The field names were re-mapped in the migration, so it shouldn't be an issue. Even so, caesius had also tested manually replacing the csv on the server with the new one, and then running the import, and he reports that he gets even more "esoteric" errors.

I'm continuing to debug this now, but we are hoping to get some additional context on this process to that end. Thank you in advance. I'll continue to update this description and provide more details as I uncover it.

caesius' original comment:

I have a migration that breaks after updating to the latest version of Migrate Source UI which I traced to this change. Does this update require updating configuration for old migrations?

What happens is that existing content gets overwritten exactly as intended by the source ID (here called MatchKey), however items new to an import do not get imported due to this error:

Migration failed with source plugin exception: 'MatchKey' is defined as a source ID but has no value.

I believe removing the Migrate Source UI version of MigrateBatchExecutable causes the migrations to use the Migrate Tools version of the same class, which for whatever reason does not work with the code or configuration used for our migrations. We're using the latest version of Migrate Tools.

I looked through the release history and issue queue for Migrate Tools but couldn't find any indication of what updates need to be done.

I'll try to debug this more and may split off a new issue to the effect of "Document this breaking change" but wanted to flag this to the people who worked on this update.

πŸ› Bug report
Status

Active

Version

1.1

Component

Code

Created by

πŸ‡ΊπŸ‡ΈUnited States bburg Washington D.C.

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @bburg
  • πŸ‡ΊπŸ‡ΈUnited States bburg Washington D.C.
  • πŸ‡ΊπŸ‡ΈUnited States bburg Washington D.C.

    So after some investigation, I think this is all that's really needed is adding this in MigrateSourceUiForm::submitForm(): to pass the updated configuration with a reference to the new file while instantiating the migration instance instead of the passing the config to MigrateBatchExecutable

        $migration = $this->pluginManagerMigration->createInstance($migration_id, $options['configuration']);
    

    Patch attached. Full disclosure, this was developed with assistance from ChatGPT.

  • πŸ‡ΊπŸ‡ΈUnited States bburg Washington D.C.
  • πŸ‡ΊπŸ‡ΈUnited States bburg Washington D.C.

    So patch in #3 does not actually fix the issue. I may have inadvertently replaced the original csv I was testing with.

  • πŸ‡ΊπŸ‡ΈUnited States nbucknor

    I'm also having this issue with version 1.1. Rolling back to 1.0 resolved the issue for me.

Production build 0.71.5 2024