Dispatch new events while importing data

Created on 27 June 2016, almost 8 years ago
Updated 17 May 2024, about 1 month ago

Problem/Motivation

It would be great to have new events getting dispatched while importing data to have better controls and modularity.
For example, if an exception occurs, the MigrateEvents::POST_IMPORT is never called, which can lead to confusion when we want to have pre and post processes to our migration in any cases.

I suggest to add a new event MigrateEvents::IMPORT_FAILED which would be called before any return MigrationInterface::RESULT_FAILED;.

In the same way, we could add a new event MigrateEvents::ROW_SKIPPED which would be called in the catch (MigrateSkipRowException $e) block.

This will allow you to add post treatments when an error occurs in your migrations, like a custom logging or archiving system, to help the user to debug and fix his migration.

Steps to reproduce

Proposed resolution

I suggest to add a new event MigrateEvents::IMPORT_FAILED which would be called before any return MigrationInterface::RESULT_FAILED;.

In the same way, we could add a new event MigrateEvents::ROW_SKIPPED which would be called in the catch (MigrateSkipRowException $e) block.

Remaining tasks

Review
Commit
Smile

User interface changes

N/A

API changes

Data model changes

Release notes snippet

✨ Feature request
Status

Needs work

Version

11.0 πŸ”₯

Component
MigrationΒ  β†’

Last updated about 19 hours ago

Created by

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • Pipeline finished with Failed
    3 months ago
    Total: 191s
    #116151
  • Pipeline finished with Success
    3 months ago
    Total: 515s
    #116154
  • Issue was unassigned.
  • Status changed to Needs review 3 months ago
  • achap πŸ‡¦πŸ‡Ί

    6 years later and a need for this came up again.

    I think I have addressed most of the issues from #19. I added additional tests for everywhere the migration fails and ensure an event is dispatched. In addition, I passed the original exception to the event since in a few cases MigrateExecutable catches the generic Exception event, which means it will catch all exceptions giving classes further up the stack no chance to react to the event. In my case, a source plugin can throw a SuspendQueueException but this was being caught by MigrateExecutable and not the queue runners. So I needed to re-throw it in my event subscriber.

    Removed the additional ideas from the issue summary as they are outside the scope of the core issue. I guess they can be opened in new issues if they're still wanted.

  • Status changed to Needs work 3 months ago
  • πŸ‡ΊπŸ‡ΈUnited States smustgrave

    Left some feedback, mostly small stuff.

    Will still need a change record.

  • Pipeline finished with Failed
    3 months ago
    Total: 2847s
    #124721
  • Pipeline finished with Failed
    3 months ago
    Total: 2183s
    #124736
  • Pipeline finished with Success
    3 months ago
    Total: 574s
    #124763
  • Status changed to Needs review 3 months ago
  • achap πŸ‡¦πŸ‡Ί

    Have added a change record and made some adjustments based on the comments and also made the getRow method consistent with regards to type hinting/annotations and the rest of the code. Will wait and see regarding the annotations issue.

  • πŸ‡ΊπŸ‡ΈUnited States smustgrave

    smustgrave β†’ changed the visibility of the branch 11.x to hidden.

  • πŸ‡ΊπŸ‡ΈUnited States smustgrave

    smustgrave β†’ changed the visibility of the branch drupal-2756269-10.1.x to hidden.

  • πŸ‡ΊπŸ‡ΈUnited States smustgrave

    smustgrave β†’ changed the visibility of the branch drupal-2756269-10.2.x to hidden.

  • Status changed to Needs work about 2 months ago
  • πŸ‡ΊπŸ‡ΈUnited States smustgrave

    Cleaning up patches and unused branches

    Sorry this fell off my radar. Everything is looking really good but think the annotations now need to be replaced with attributes. Will keep an eye out for this one.

  • Pipeline finished with Failed
    about 1 month ago
    Total: 214s
    #173881
  • Pipeline finished with Failed
    about 1 month ago
    Total: 595s
    #173901
  • Pipeline finished with Failed
    about 1 month ago
    Total: 5287s
    #173910
  • Pipeline finished with Failed
    about 1 month ago
    Total: 607s
    #174834
  • Pipeline finished with Failed
    about 1 month ago
    Total: 523s
    #174844
  • achap πŸ‡¦πŸ‡Ί

    I've converted those annotations to php attributes. Unfortunately when I've merged in the latest code from 11.x there is a test failure in testRowSkippedEvent however the error message is obscured by the annoying "Exception: Serialization of 'Closure' is not allowed" message which doesn't actually show what's wrong. When I run MigrateEventsTest.php locally I don't get any errors so this one is hard to debug. Apparently this is quite a common issue faced in Migrate tests (see related issue).

    I do have a stack trace:

        /builds/issue/drupal-2756269/core/lib/Drupal/Core/Cache/MemoryBackend.php:119
        /builds/issue/drupal-2756269/core/lib/Drupal/Core/Cache/CacheCollector.php:275
        /builds/issue/drupal-2756269/core/lib/Drupal/Core/State/State.php:102
        /builds/issue/drupal-2756269/core/modules/migrate/tests/src/Kernel/MigrateEventsTest.php:481
        /builds/issue/drupal-2756269/vendor/symfony/event-dispatcher/EventDispatcher.php:206
        /builds/issue/drupal-2756269/vendor/symfony/event-dispatcher/EventDispatcher.php:56
        /builds/issue/drupal-2756269/core/modules/migrate/src/MigrateExecutable.php:247
        /builds/issue/drupal-2756269/core/modules/migrate/tests/src/Kernel/MigrateEventsTest.php:354
    

    That shows that it is trying to serialize the cached data (which would contain an exception). My guess would be that the exception data has a closure in there somewhere but like I said very hard to debug if I can't reproduce it locally.

    Hopefully someone can help pinpoint the issue!

Production build 0.69.0 2024