Migrate support for deleting items no longer in the incoming data

Created on 30 September 2016, about 8 years ago
Updated 10 July 2024, 6 months ago

Problem/Motivation

We would like to add support for deleting items that are no longer present in the incoming data when a migration is run. This was a feature of Feeds in D7 and seems like a common need.

Proposed resolution

From @mikeryan in #15:

I really think this should be a distinct "purge" operation that gathers the currently-available source IDs from the source plugin, and scans the map table for any which aren't in that list to invoke the destination rollback on them. Now that I think about it, maybe it could even be an option on rollback - drush migrate-rollback --missing-from-source my_migration.

Completed tasks

Two tests added and passing tests.

  • MigrateRollbackMissingTest - is this sufficient? it does not test the source all_row property.
  • MigrateRollbackMissingMessageTest

Remaining tasks

  • Decide whether a simpler approach will work. See the discussion in Comments #133, #135, #136, #138.
  • If we stay with the current approach, then address the points raised in Comment #139.
  • Manual testing. See instructions in the next section.
  • Add documentation.
  • Add a change record.

Manual testing

At this point, testing is a little complicated. The current patch (in Comment #136) adds an interface and a trait. In order for selective rollback to work, your migration source plugin needs to implement the interface. An easy way to do that is by using the trait. For examples, see the code snippet below the following steps.

  1. Install Migrate Plus, at least version 8.x-4.2, for #3045346: Add missing source item event .
  2. Apply the current patch from this issue.
  3. Apply the current patch from Implement rollback of items no longer in source data Closed: duplicate .
  4. Create a source plugin that implements the new SyncableSourceInterface interface.
  5. Update your migrations to use your custom source plugin.
  6. Clear caches.
  7. You can now roll back your migration with the new option provided by Implement rollback of items no longer in source data Closed: duplicate : drush mr my_migration --missing-from-source.

For Step 4, you can follow the example in the SyncableEmbeddedDataSource plugin that is added in this issue. That class is used in the new MigrateRollbackMissingTest test class. Here is a class that @marvil07 wrote for a client project that uses the patch from this issue:


namespace Drupal\my_migrate\Plugin\migrate\source;

use Drupal\migrate\Plugin\MigrationInterface;
use Drupal\migrate_plus\Plugin\migrate\source\Url;
use Drupal\migrate_tools\Plugin\migrate\source\SyncableSourceTrait;
use Drupal\migrate_tools\SyncableSourceInterface;

/**
 * A syncable url source using SyncableSourceTrait.
 *
 * @see Drupal\migrate_plus\Plugin\migrate\source\Url
 *
 * @MigrateSource(
 *   id = "syncable_url",
 *   source_module = "my_migrate"
 * )
 */
class SyncableUrl extends Url implements SyncableSourceInterface {

  use SyncableSourceTrait;

  /**
   * {@inheritdoc}
   */
  public function __construct(array $configuration, $plugin_id, $plugin_definition, MigrationInterface $migration) {
    parent::__construct($configuration, $plugin_id, $plugin_definition, $migration);
    $this->setAllRowsFromConfiguration();
  }

}

User interface changes

N/A

API changes

Data model changes

N/A

Feature request
Status

Fixed

Version

4.0

Component

Code

Created by

🇺🇸United States damontgomery

Live updates comments and jobs are added and updated live.
  • Contributed project blocker

    It denotes an issue that prevents porting of a contributed project to the stable version of Drupal due to missing APIs, regressions, and so on.

  • Needs manual testing

    The change/bugfix cannot be fully demonstrated by automated testing, and thus requires manual testing in a variety of environments.

Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • 🇺🇸United States ddavisboxleitner

    In 9.1 the order of the arguments to dispatch() changed, and so the previous patches stop working in those higher version of Drupal. In testing, just re-ordering the arguments in all calls to dispatch() works to restore this feature to functioning.

  • 🇵🇹Portugal developer-rocha

    I've updated the latest patch with @ddavisboxleitner's comment about dispatch.

Production build 0.71.5 2024