Update MigratePluginManager to include both attribute and annotation class

Created on 29 February 2024, 9 months ago
Updated 19 April 2024, 7 months ago

Problem/Motivation

Drupal\migrate\Plugin\MigratePluginManager is the base class for plugin managers for 4 different migrate plugin types: MigrateSource, MigrateDestination, MigrateProcessPlugin, and MigrateField. In order to convert the 4 plugin types to use attributes, MigratePluginManager needs to be updated first.

This is exploring the changes required for Drupal\migrate\Plugin\MigratePluginManager and children.

Proposed resolution

  1. Update Drupal\migrate\Plugin\MigratePluginManager constructor to include attribute and annotation class names

The plugins converted here are

  • Book destination plugin
  • DateField field plugin
  • EntityFile destination plugin
  • Explode process plugin
  • NullIdMap
  • Sql

Remaining tasks

User interface changes

API changes

Data model changes

Release notes snippet

πŸ“Œ Task
Status

Fixed

Version

10.3 ✨

Component
MigrationΒ  β†’

Last updated 1 day ago

Created by

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @godotislate
  • Pipeline finished with Success
    9 months ago
    Total: 642s
    #106594
  • Status changed to Needs review 9 months ago
  • Put up MR with one approach to tackle BC for migrate plugin manager.

  • First commit to issue fork.
  • πŸ‡³πŸ‡ΏNew Zealand quietone

    Let's test this by converting one process plugin, 'explode'.

  • Pipeline finished with Failed
    9 months ago
    Total: 518s
    #106732
  • Pipeline finished with Success
    9 months ago
    Total: 1293s
    #106811
  • Status changed to Needs work 9 months ago
  • πŸ‡ΊπŸ‡ΈUnited States smustgrave

    Left a small change and one question.

  • Pipeline finished with Success
    9 months ago
    Total: 511s
    #107095
  • Status changed to Needs review 9 months ago
  • Status changed to RTBC 9 months ago
  • πŸ‡ΊπŸ‡ΈUnited States smustgrave

    Thanks for answering that for me @godotislate

  • Status changed to Needs work 9 months ago
  • πŸ‡³πŸ‡ΏNew Zealand quietone

    Thanks for the reviews!

    I think this should include the conversion of a source, destination and field plugin as well as discussed in Slack.https://drupal.slack.com/archives/C1BMUQ9U6/p1709180072609059. Lets make sure that all plugin types that use this manager can work.

  • Pipeline finished with Failed
    9 months ago
    Total: 570s
    #107490
  • Pipeline finished with Success
    9 months ago
    Total: 496s
    #107500
  • Status changed to Needs review 9 months ago
  • Originally pushed conversion for at least attribute of each type, but tests failed on the source plugin. I cherry-picked from MR 6780 for πŸ“Œ Convert MigrateSource plugin discovery to attributes Needs work , and it looks like work in that issue for the Attribute class is incomplete, including an outstanding question on how to handle multiple provders. So I reverted the changes for Migrate source, and test are passing for one plugin of each of the other three.

  • Status changed to Needs work 9 months ago
  • πŸ‡³πŸ‡ΏNew Zealand quietone

    Migrate source plugins have their own discovery using, \Drupal\migrate\Plugin\Discovery\AnnotatedClassDiscoveryAutomatedProviders and they also have to handle multiple providers, \Drupal\migrate\Annotation\MultipleProviderAnnotationInterface. I am not familiar with those and am just starting to take a look.

  • Pipeline finished with Failed
    9 months ago
    Total: 524s
    #109300
  • Pipeline finished with Failed
    9 months ago
    Total: 182s
    #109321
  • Pipeline finished with Failed
    9 months ago
    #109332
  • Pipeline finished with Success
    9 months ago
    Total: 510s
    #109864
  • Pipeline finished with Success
    9 months ago
    #109897
  • πŸ‡³πŸ‡ΏNew Zealand quietone
  • Status changed to Needs review 9 months ago
  • πŸ‡³πŸ‡ΏNew Zealand quietone

    This may need to be split up but I did find it helpful to work on all these at one time. The most problematic is the source plugins because it requires multiple providers which meant changes to discovery. And adding a class property, $providers.

  • Status changed to Needs work 9 months ago
  • A couple comments/questions on the MR.

    Also, I wonder if the work to deal with sources having multiple providers shouldn't be done in πŸ“Œ Convert MigrateSource plugin discovery to attributes Needs work , so iterating on that part doesn't block getting the other plugin types done.

  • First commit to issue fork.
  • Pipeline finished with Canceled
    9 months ago
    Total: 312s
    #111523
  • Status changed to Needs review 9 months ago
  • πŸ‡·πŸ‡ΊRussia sorlov

    fixed feedbacks from MR

  • Pipeline finished with Success
    9 months ago
    Total: 490s
    #111530
  • Pipeline finished with Success
    9 months ago
    Total: 2522s
    #111573
  • πŸ‡³πŸ‡ΏNew Zealand quietone

    Add the two generic plugins that also need the MigratePluginManager

  • πŸ‡³πŸ‡ΏNew Zealand quietone

    Added commas to meet Coding Standards per Function Declarations β†’ , which is a new addition to the standards.

  • πŸ‡ΊπŸ‡ΈUnited States smustgrave

    Left 1 question but will leave in review for additional eyes. Based on other attribute tickets change look good to me.

  • Status changed to Needs work 8 months ago
  • πŸ‡ΊπŸ‡ΈUnited States benjifisher Boston area

    I reviewed MR 6827. I have a bunch of minor suggestions for code cleanup, which I will add to the MR as suggestions. This is the first time that I have looked at attributes for plugin definitions, so take my suggestions with a grain of salt. (That is, do not be surprised if some suggestions seem like bad ideas.)

    This MR has conflicts with πŸ“Œ Convert the PHPStan baseline from NEON to PHP Fixed . One commit adds an exception to the PHPStan baseline, and a later commit removes that exception. If you remove those two changes, then you should be able to rebase cleanly.

  • Pipeline finished with Failed
    8 months ago
    Total: 353s
    #121240
  • Pipeline finished with Failed
    8 months ago
    Total: 180s
    #121246
  • Pipeline finished with Success
    8 months ago
    Total: 716s
    #121250
  • Pipeline finished with Failed
    8 months ago
    Total: 675s
    #121279
  • Pipeline finished with Success
    8 months ago
    Total: 566s
    #121294
  • Pipeline finished with Failed
    8 months ago
    Total: 188s
    #121305
  • Pipeline finished with Failed
    8 months ago
    Total: 180s
    #121307
  • Pipeline finished with Failed
    8 months ago
    Total: 188s
    #121310
  • Pipeline finished with Success
    8 months ago
    Total: 598s
    #121318
  • Pipeline finished with Success
    8 months ago
    Total: 577s
    #121655
  • Status changed to Needs review 8 months ago
  • I think all MR feedback has been addressed with changes or commented on.

  • πŸ‡ΊπŸ‡ΈUnited States benjifisher Boston area

    @godotislate:

    Thanks. I am not sure when I will have time to re-review. Also, I am not sure I am qualified to declare the issue RTBC. As I said, this is the first time I have looked at attributes.

    Do we need additional test coverage, or are the existing tests good enough now that we have converted a few plugins?

    I think we should at least add a test to confirm that the system works with a combination of annotation and attribute plugins. Once we convert Drupal core to attributes, there will still be contrib modules using annotations.

  • Status changed to Needs work 8 months ago
  • Do we need additional test coverage, or are the existing tests good enough now that we have converted a few plugins?

    I think we should at least add a test to confirm that the system works with a combination of annotation and attribute plugins. Once we convert Drupal core to attributes, there will still be contrib modules using annotations.

    This thought did occur to me. We think things are fine now, because in the MR we have a few plugins converted to attributes and most of the rest left with annotations, with all tests passing. But once all the core annotations are removed, annotation support for migration multiple providers might not be directly tested anymore. Looks like for other plugin types there is Drupal\Tests\Component\Annotation\AnnotatedClass\DiscoveryTest, so we'll need something similar for migration source plugins.

  • Pipeline finished with Failed
    8 months ago
    #124483
  • Pipeline finished with Failed
    8 months ago
    Total: 461s
    #124517
  • Pipeline finished with Failed
    8 months ago
    Total: 702s
    #124554
  • Pipeline finished with Failed
    8 months ago
    Total: 495s
    #124568
  • Pipeline finished with Success
    8 months ago
    Total: 573s
    #124573
  • Status changed to Needs review 8 months ago
  • Added the test for backwards compatibility for annotated migrate source plugins.

  • Status changed to RTBC 8 months ago
  • πŸ‡ΊπŸ‡ΈUnited States smustgrave

    Appears all feedback has been addressed. Went through one more time and I don't have any questions

  • Status changed to Needs work 8 months ago
  • πŸ‡¬πŸ‡§United Kingdom alexpott πŸ‡ͺπŸ‡ΊπŸŒ

    Added some comments to the MR.

  • Pipeline finished with Success
    8 months ago
    Total: 626s
    #129867
  • Pipeline finished with Success
    8 months ago
    #129890
  • Addressed most comments. Left questions open about requirements_met functionality for anyone more familiar with that than me.

  • Status changed to Needs review 8 months ago
  • πŸ‡³πŸ‡ΏNew Zealand quietone

    Changing status for another review.

    Also, it needs to be decided if all the migrate plugins will be converted in this issue. This issue is already rather large, 37 files + 763 βˆ’ 105.

  • πŸ‡³πŸ‡ΏNew Zealand quietone

    To help with the decision there are:

    • 118 MigrateSource plugins
    • 24 MigrateFielld plugins
    • 79 MigrateProcessPlugins plugins
    • 31 MigrateDestination plugins
  • That seems like a lot of additional files to look at. I think those can be done in the original migration plugin type conversion tickets and reviewed separately.

  • πŸ‡ΊπŸ‡ΈUnited States smustgrave

    Is there any pro to combining? Seems like keeping separate makes the most sense from a reviewing standpoint.

  • πŸ‡¬πŸ‡§United Kingdom catch

    I think it would be fine to do those in the four existing issues that are postponed on this one.

  • πŸ‡ΊπŸ‡ΈUnited States smustgrave

    Sweet, looking at the MR I believe all threads have been answer, there any outstanding questions?

  • Status changed to RTBC 8 months ago
  • πŸ‡ΊπŸ‡ΈUnited States smustgrave

    Going to go ahead and mark it as I didn't see any outstanding threads

  • πŸ‡¬πŸ‡§United Kingdom alexpott πŸ‡ͺπŸ‡ΊπŸŒ

    Committed and pushed c9d53fd473 to 11.x and d7f89942a7 to 10.3.x. Thanks!

    I've had a good look and the changes to the plugin managers to support multiple providers and I think this is the best we can do. It would be lovely to move this upstream from migrate to core somehow.

    • alexpott β†’ committed d7f89942 on 10.3.x
      Issue #3424509 by godotislate, quietone, sorlov, smustgrave, benjifisher...
  • Status changed to Fixed 8 months ago
    • alexpott β†’ committed c9d53fd4 on 11.x
      Issue #3424509 by godotislate, quietone, sorlov, smustgrave, benjifisher...
  • πŸ‡³πŸ‡ΏNew Zealand quietone

    The issue to move the support for multiple providers to the plugin system is #2786355: Move multiple provider plugin work from migrate into core's plugin system β†’

    • quietone β†’ committed 580bfcb7 on 11.x
      Revert "Issue #3424509 by godotislate, quietone, sorlov, smustgrave,...
    • quietone β†’ committed effc39d1 on 10.3.x
      Revert "Issue #3424509 by godotislate, quietone, sorlov, smustgrave,...
  • Status changed to Needs work 8 months ago
  • πŸ‡³πŸ‡ΏNew Zealand quietone

    Reverted because this was requiring source_module on all source plugins, when it is really just for migrate drupal.

  • Pipeline finished with Failed
    8 months ago
    Total: 180s
    #137024
  • πŸ‡³πŸ‡ΏNew Zealand quietone

    I remembered πŸ“Œ Move source_module and destination_module from Migrate to Migrate Drupal Needs work . So I have updated that and it looks like it will help with the problem of the source_module.

  • Pipeline finished with Success
    8 months ago
    Total: 634s
    #137478
  • Status changed to Needs review 8 months ago
  • I think the best way to keep moving forward is to de-scope everything to do with migrate source plugins from this issue and handle that in πŸ“Œ Convert MigrateSource plugin discovery to attributes Needs work . The remainder of the reverted commit for this issue can be brought back here.

    I've put up MR 7327 for the proposed changes for this issue:

    • MigratePluginManager
    • Destination, Process, and Field attributes
    • Sample destination, process, and field plugins
    • Doc block references to destination, process, and field annotation classes

    Remaining work to do in in πŸ“Œ Convert MigrateSource plugin discovery to attributes Needs work :

    • Automated/multiple provider handling
    • MigrateSourceManager
    • Source plugins
    • Source attribute
    • Docblock references to source annotations

    A lot of the migrate source work will just be restoring the remaining diff from the reverted commit. Along with dealing with the source_module property in the attribute, and then converting the remainder of the plugins.

  • Status changed to RTBC 7 months ago
  • πŸ‡ΊπŸ‡ΈUnited States smustgrave

    I'm in favor of breaking out more. Makes reviews simpler and know we shouldn't assume anything but I can put money the other postponed tickets will be addressed before 11.x release, they got all the attention right now haha.

    So looking at MR 7327 tests are green from the re-scope.

  • πŸ‡¬πŸ‡§United Kingdom alexpott πŸ‡ͺπŸ‡ΊπŸŒ

    Committed and pushed 7dd6b190eb to 11.x and b7e0c8e94b to 10.3.x. Thanks!

    Great idea to get most of this done.

    • alexpott β†’ committed b7e0c8e9 on 10.3.x
      Issue #3424509 by godotislate, quietone, sorlov, smustgrave, alexpott,...
  • Status changed to Fixed 7 months ago
    • alexpott β†’ committed 7dd6b190 on 11.x
      Issue #3424509 by godotislate, quietone, sorlov, smustgrave, alexpott,...
  • πŸ‡¬πŸ‡§United Kingdom alexpott πŸ‡ͺπŸ‡ΊπŸŒ

    I've updated all the all issues postponed on this one.

  • Automatically closed - issue fixed for 2 weeks with no activity.

Production build 0.71.5 2024