Drupal 10 support

I classify myself as an Ambitious Site Builder who's returning to Drupal 8+ after developing D7 sites exclusively. I'm seeking guidance on how to begin testing this on Drupal 10.

WELCOME! 😄👋

I'm seeking guidance on how to begin testing this on Drupal 10.

Thank you for your kind offer!

I've proceeded successfully through a D7 to D9 AM:A migration with a mildly complex site (Data imported: 15922/15944 - the remaining 17 errors were of no consequence).

Glad to hear this! 😊

Add a command line parameter to generate either a D9 or D10 site. This option would help site builders troubleshoot blockers to D10 compatibility. Possibly add a disclaimer of associated risks if "d10" is chosen because not all modules, patches, or solutions have been vetted.

That'd be the dream indeed. But it's not that simple … we'd have to create a separate set of recommendations. Because many of the current recommendations will only ever work for Drupal 9, since not all modules have Drupal 9 equivalents!

Also see the FAQ entry above the one you linked:

• Why Drupal 9 and not 10?

  See the part above about manually vetting migrations, which resulted in "vetted migrations". Unfortunately, PHP 8 behaves differently and Drupal 10 requires PHP 8.1. Plus, there are still more modules available for Drupal 9 than for 10, so the migration would be less complete.
  
  Fortunately, thanks to us Drupal having made upgrades easy forever (since Drupal 8), migrating from 7 to 9 is 90% of the work. From 9 to 10 is trivial in comparison.

Please advise on how I would begin to test this with Drupal 10.

Let me think about this for a while 🤔

High-level plan

• acli app:new:from:drupal7 defaults to the latest recommendations.
• … but by doing acli app:new:from:drupal7 --recommendations /path/to/local/recommendations.json or acli app:new:from:drupal7 --recommendations https://example.com/recommendations.json, you can tell it to use different recommendations. That is how the test coverage has been able to run tests locally, without fetching the live latest recommendations 😊
• https://git.drupalcode.org/project/acquia_migrate/-/tree/recommendations are the current canonical recommendations. We could create a new recommendations-10 branch. People could then manually use this by doing acli app:new:from:drupal7 --recommendations https://git.drupalcode.org/project/acquia_migrate/-/raw/recommendations-10/recommendations.json.
• Once the module test suite is passing against both recommendations and recommendations-10, we could create an alias in the ACLI command itself, allowing the magic string 10 to automatically expand into the aforementioned URL.
• Eventually, if/when the Drupal 10 recommendations match or surpass the Drupal 9 ones, we would rename the current recommendations branch to recommendations-9, and renamerecommendations-10 to recommendations.

What would it take to get a recommendations-10 branch going?

This branch would have to start from scratch:

1. Hard required first step: Drupal 10 core (and port all patches) + Media Migration in an otherwise empty curated.json
2. Get the entire recommendations test suite (but ported to recommendations-10) + module test suite (against both recommendations and recommendations-10 simultaneously) to pass.
3. Port the other recommendations in curated.json. Start with the most impactful module migrations, which do have explicit contrib migration test coverage in the module test suite: @group acquia_migrate__contrib.
4. Generate a D10 equivalent of generated.json by adding a new parameter to the crawl.php script, to allow running it for either D9 or D10

The first 2 steps must happen in a single issue (with 2 MRs: one for the recommendations, one for the module — see 📌 Update: drupal/core:9.5.11 Fixed for an example of that pattern). Once both MRs are green, I will convert the recommendations one to the new recommendations-10 branch, and the module one I will merge into the module.

From that point onward, we can gradually grow the recommendations-10 branch. Once it's grown to make the last bullet of the high-level plan a reality, we can rename the branches/change the defaults.

I finally just got around to reading https://ddev.com/blog/drupal7-drupal9-migration-ddev-acquia-migrate-acce... — and OH MY THIS IS YOU!!!!!! 😄

Thanks for that amazing write-up — I'm certain that your write-up will help dozens if not thousands of Drupal users (and not just site builders: my ddev- or infra-fu in general is not strong because I never have to deal with it: this would have helped me if I hadn't built AM:A! 🙏).

Thank you! 😊

P.S.: last time I looked, https://www.drupal.org/project/views_migration → yielded such imperfect results that it seemed worse than manually recreating views — and usually in such a big transition you'd want to reassess them anyway. Still, point taken!
P.P.S.: https://www.drupal.org/project/page_manager_migration → is a mere PoC, built by the team that built AM:A (the amazing @huzooka!) — it did not meet the quality gate to install it by default.

The bare Drupal 10 recommendations branch required a LOT of rerolls today (despite preliminary work to do this in December 2022):

• #3096972-71: The Drupal 7 ThemeSettings source plugin does not check that the destination site has a valid theme to migrate settings into →
• #3097336-91: Improve the DX for migrating content entities: view modes, fields, formatters, widgets etc should be migrated per entity type + bundle →
• #3115938-24: Derive block migration per theme and per block plugin type →
• #3123775-19: Ensure that migrations of entities which have file or image fields depend on private/public files migration →

… but it now results in an Drupal 10.2 AM:A project! 🥳 Next up:

1. use this branch to test the Drupal 10 compatibility of the module
2. make the module Drupal 10-compatible
3. use the Drupal 10-compatible module to fully test the recommendations

This is so bizarre.

I can apply the entire core patch stack just fine like this:

$ git reset --hard 10.2.0 && sh apply-patches.sh drupal/core modules/contrib/acquia_migrate/curated.json

… but if I do the same thing using composer install I get:

  - Installing chi-teck/drupal-code-generator (3.3.0): Extracting archive
  - Installing drush/drush (12.4.3): Extracting archive
  - Applying patches for drupal/core
    https://www.drupal.org/files/issues/2021-02-24/core-derive_path_alias_migrations-3122649-41.patch ([AMA_MIGRATE_FIX] Issue #3122649: Derive path alias migrations per entity type (and bundle))
    https://www.drupal.org/files/issues/2024-01-04/core-theme_settings_migrate_requirement-3096972-71-10.2.0.patch ([AMA_MIGRATE_FIX] Issue #3096972: The Drupal 7 ThemeSettings source plugin does not check that the destination site has a valid theme to migrate settings into)
    https://www.drupal.org/files/issues/2023-08-17/3204212-field-migration-widget-formatter-mapping-58-D11--fix-only.patch ([AMA_MIGRATE_FIX] Issue #3204212: Convert remaining widget and formatter type migrations to MigrateField plugins)
    https://www.drupal.org/files/issues/2022-12-20/core-allow_map_formatter_migration-3202462-20--on-top-of-3204212-52.patch ([AMA_MIGRATE_FIX] Issue #3202462: Provide option for contrib modules to map their D6 / D7 field formatter and widget plugin IDs to the equivalent D9 plugin ID)
    https://www.drupal.org/files/issues/2021-06-10/core-migrate_field_formatter_widget_with_fallback-3108302-45--on-top-of-3202462-8_0.patch ([AMA_MIGRATE_FIX] Issue #3108302: Field formatter & widget settings: fall back to default if missing plugin)
    https://www.drupal.org/files/issues/2024-01-04/core-derived_field_and_field_display_migrations-3097336-91-10.2.0.patch ([AMA_MIGRATE_FIX] Issue #3097336: Improve the DX for migrating content entities: view modes, fields, formatters, widgets etc should be migrated per entity type + bundle)
   Could not apply patch! Skipping. The error was: Cannot apply patch https://www.drupal.org/files/issues/2024-01-04/core-derived_field_and_field_display_migrations-3097336-91-10.2.0.patch
    https://www.drupal.org/files/issues/2022-12-22/3198732-33.patch ([AMA_MIGRATE_FIX] Issue #3198732: Migrating reference fields: target_bundles may never be empty array)
    https://www.drupal.org/files/issues/2024-01-04/core-derived_block_config_migrations-3115938-24-10.2.0.patch ([AMA_MIGRATE_FIX] Issue #3115938: Derive block migration per theme and per block plugin type)
    https://www.drupal.org/files/issues/2024-01-04/core-add_file_migration_dependencies-3123775-19--10.2.0--fix-only--do-not-test.patch ([AMA_MIGRATE_FIX] Issue #3123775: Ensure that migrations of entities with file or image fields depend on private/public files migration)
    https://www.drupal.org/files/issues/2022-12-20/2845340-63-fix-only-do-not-test.patch ([AMA_MIGRATE_FIX] Issue #2845340: migrate mapping & messages table names are truncated, can lead to incorrect mapping lookups)
    https://www.drupal.org/files/issues/2020-12-07/3151979-22.patch ([AMA_MIGRATE_FIX] Issue #3151979: System file settings migration (d6_system_file and d7_system_file) assumes that allow_insecure_uploads variable is always set)
    https://www.drupal.org/files/issues/2020-07-01/core-route_migrate_process_plugin_options-3156083-2--complete.patch ([AMA_MIGRATE_FIX] Issue #3156083: Route migrate process plugin shouldn't assume that the $options variable is always an array)
    https://www.drupal.org/files/issues/2023-03-16/core-derive_menu_link_migrations_per_entity_type-3051251-69.patch ([AMA_MIGRATE_FIX] Issue #3051251: Existing menu links show validation issues on migration)
    https://www.drupal.org/files/issues/2020-06-23/3154156-2.patch ([AMA_MIGRATE_FIX] Issue #3154156: Improve migration system performance: statically cache DrupalSqlBase::$systemData)
    https://www.drupal.org/files/issues/2024-01-04/core-create_stub_only_when_matching_source_row_exists-3156730-57--10.2.x.patch ([AMA_MIGRATE_FIX] Issue #3156730: Stubs should only be created if the referenced source row actually exists)
    https://www.drupal.org/files/issues/2022-03-14/3156733-15.patch ([AMA_MIGRATE_FIX] Issue #3156733: File migration's "owner" user reference should use migration_lookup)
    https://www.drupal.org/files/issues/2020-08-18/3165813-2.patch ([AMA_MIGRATE_FIX] Issue #3165813: Undefined index: text_processing in Drupal\text\Plugin\migrate\field\d7\TextField)
    https://www.drupal.org/files/issues/2022-12-22/3166930-34.patch ([AMA_MIGRATE_FIX] Issue #3166930: Migrated filters may have invalid/incomplete settings applied)
    https://www.drupal.org/files/issues/2021-11-04/drupal-3167267-36.patch ([AMA_MIGRATE_FIX] Issue #3167267: MigrateExecutable should catch not only exceptions, but also fatal errors)
    https://www.drupal.org/files/issues/2020-12-08/core-content_entity_source_exception_when_bundle_is_missing-3186449-2.patch ([AMA_MIGRATE_FIX] Issue #3186449: ContentEntity source plugin shouldn't throw exception when the bundle key is missing)
    https://www.drupal.org/files/issues/2020-12-09/core-nodecomplete_wrong_source_langcode-3187419-2.patch ([AMA_MIGRATE_FIX] Issue #3187419: d7/NodeComplete source plugin adds invalid source "source_langcode" for "content_translation_source" destination property)
    https://www.drupal.org/files/issues/2020-12-14/core-improve_source_record_count_i18n_string-3187474-9.patch ([AMA_MIGRATE_FIX] Issue #3187474: Improve source record count of translation migrate source plugins which use the "i18n_string" table)
    https://www.drupal.org/files/issues/2021-06-10/2859314-58.patch ([AMA_MIGRATE_FIX] Issue #2859314: Highwater condition with unjoined maps skips unprocessed and NEEDS_UPDATE rows)
    https://www.drupal.org/files/issues/2021-03-01/core-allow_migrating_forward_revisions-3200949-9.patch ([AMA_MIGRATE_FIX] Issue #3200949: Unpublished entity revisions get published because EntityContentComplete)
    https://www.drupal.org/files/issues/2021-03-19/3204343-9.patch ([AMA_KEEP] Issue #3204343: Disabling the default search page can make the entire site inaccessible)
    https://www.drupal.org/files/issues/2021-12-28/core-fix_entityconfigbase-3118262-52--complete.patch ([AMA_MIGRATE_FIX] Issue #3118262: Calling EntityConfig::import() with multiple destination IDs fails)
    https://gist.githubusercontent.com/zolhorvath/9b8d28df3dd45e3d8a8234e590016aa7/raw/cba7add887d8d20564b1fed45c52ce803575b58d/core-migrate_conflicting_text_fields-3213636-6--9.1.x--fix-and-db-fixture.patch ([AMA_MIGRATE_FIX] Issue #3213636: Prevent data loss: migrate text fields with conflicting text_processing setting as formatter text fields)
    https://www.drupal.org/files/issues/2021-06-10/core-allow_altering_migrate_field_value_query-3218294-2.patch ([AMA_MIGRATE_FIX] Issue #3218294: Allow altering field value query performed by FieldableEntity)
    https://www.drupal.org/files/issues/2021-11-19/core-fix_et_mapping_of_taxonomy_terms-3219078-13.patch ([AMA_MIGRATE_FIX] Issue #3219078: Regression: multilingual taxonomy term migrations are failing if user tries to migrate from Drupal 7.81+ and Entity Translation 7.x-1.1)
    https://www.drupal.org/files/issues/2022-04-15/core-derive_statistics_module_migrations-3226744-26.patch ([AMA_MIGRATE_FIX] Issue #3226744: Derive statistics module migrations per node type)
    https://www.drupal.org/files/issues/2021-08-11/3227361-13.patch ([AMA_MIGRATE_FIX] Issue #3227361: Fix \Drupal\migrate\Plugin\migrate\source\SqlBase::initializeIterator()'s cross-database joining broken when using particular DB/table names)
    https://www.drupal.org/files/issues/2022-12-07/2329253-97.patch ([AMA_KEEP] Issue #2329253: Allow the ChangedItem to skip updating)
    https://www.drupal.org/files/issues/2022-02-21/core-search_settings_migration-3263622-11.patch ([AMA_MIGRATE_FIX] Issue #3263622: Search Settings (d7_search_settings) migration is ignored if any of the variables it would use are missing.)
    https://www.drupal.org/files/issues/2022-04-19/unset_statistics_node_tranlation_counter-3275963-2.patch ([AMA_MIGRATE_FIX] Issue #3275963: Unset Statistics node translation counter migration.)
    https://gist.githubusercontent.com/wimleers/393c12fb3e460251a5958ad681cdb2d3/raw/e29d2ce77009cc68483b007789fe4089507ab55c/standard-remove_default_content_entity_types.patch ([AMA_MIGRATE_FIX] Remove certain default content entity types (node, block_content, taxonomy_term, comment) from Drupal's Standard install profile)

  - Applying patches for drupal/media_migration
    https://www.drupal.org/files/issues/2021-07-21/media_migration-allow_map_formatter_migration-3224679-2.patch ([AMA_MIGRATE_FIX] Issue #3224679: Map D7 Amp module's amp_image formatter to D9 amp_media if the image field was migrated as entity reference)
    https://www.drupal.org/files/issues/2021-08-11/media_migration-allow_colorbox_formatter_migration-3224657-6-on_top_of-3224679-2-do_not_test.patch ([AMA_MIGRATE_FIX] Issue #3224657: Map colorbox formatters to media_colorbox)
    https://www.drupal.org/files/issues/2021-08-26/media_migration-map_mediaelements_formatters-3229931-2-on_top_of-3224657-6-do_not_test.patch ([AMA_MIGRATE_FIX] Issue #3229931: Map all Mediaelement module's formatter from Drupal 7)
    https://gist.githubusercontent.com/zolhorvath/c6e89a491fa4f142be54dcec3055c058/raw/a5cd8e27ba8302a2d8432337ad8843dd32d8904d/media_migration-token_destination_default_media_embed.patch ([AMA_MIGRATE_FIX] Change Media Migration's default token destination filter plugin ID to media_embed)

17 package suggestions were added by new dependencies, use `composer suggest` to see details.
Generating autoload files

😬😭

composer install -vvv to the rescue!

Found it:

SNIP

patching file modules/field/migrations/d7_field_instance.yml

patching file modules/field/migrations/d7_field_instance_widget_settings.yml

can't find file to patch at input line 994
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------

|diff --git a/core/modules/field/migrations/d7_field_instance_widget_settings.yml.orig b/core/modules/field/migrations/d7_field_instance_widget_settings.yml.orig
|index c371900b134..a0c52d24068 100644
|--- a/core/modules/field/migrations/d7_field_instance_widget_settings.yml.orig
|+++ b/core/modules/field/migrations/d7_field_instance_widget_settings.yml.orig
--------------------------
File to patch: 
Skip this patch? [y] 
Skipping patch.

2 out of 2 hunks ignored


patching file modules/field/migrations/d7_view_modes.yml


SNIP

— for some reason this gets treated differently. Rerolling #3097336-91: Improve the DX for migrating content entities: view modes, fields, formatters, widgets etc should be migrated per entity type + bundle → ought to be enough 👍

In principle, composer config repositories.foo path ~/core/modules/contrib/acquia_migrate should be enough to register a path repo that "wins" from the official d.o packagist facade.

In practice, I can't get it to work. 😬 I tried literally everything in https://stackoverflow.com/questions/36745691/cant-get-composer-path-repo....

Will continue that tomorrow.

The D10 recommendations MR is now green! It doesn't quite make sense yet to convert this into the new recommendations-10 branch. Because it will only start working once there is a Drupal 10-compatible module branch too.

This is a bit of a 🐥 vs 🥚 situation though: both must be created simultaneously, because both refer to each other.

Once the module CI pipeline is also green (and I'm getting close), I will create the following branches:

1. recommendations-10 (new branch, independent from recommendations)
2. 1.9.x (continued from 1.8.x, which will be marked unsupported)

While I'm working to get the module CI pipeline to get to green too, I'm also running the script to update generated.json, but that takes hours, so I'll just let that run overnight :)

Note to self:

$ node -r dotenv-safe/config ./node_modules/.bin/nightwatch --config ./tests/Drupal/Nightwatch/nightwatch.conf.js --tag=acquia_migrate
  Error
   No tests defined! using source folder: ../core/modules/ckeditor5/tests/src/Nightwatch/Tests,../core/modules/toolbar/tests/src/Nightwatch/Tests,../core/tests/Drupal/Nightwatch/Tests,../modules/contrib/decoupled_pages/tests/src/Nightwatch/Tests
    - using tags filter: acquia_migrate

👆 Some things changed in Nightwatch testing in D10, this will need to be updated in the test infra most likely.

Whew, finally figured out why the mapping override tests were failing: #3413533-5: Fix failing tests on HEAD → — a regression caused by 📌 Manually clear cache keys from plugin managers with finite variations instead of using cache tags Fixed .

For @dan612's MR remark from yesterday, I created 📌 Refine the crawler that generates `generated.json` to support full Composer constraint expressions Active .

75bb1fe3de81182a6d206c2fb3938192a627b1f4 (HEAD -> 3399733-drupal-10-ci) Remove work-arounds and instead require `migrate_plus:^6.0.2`, which shipped yesterday!
🥳

Thanks to https://www.drupal.org/project/migrate_plus/releases/6.0.2 → , which I pinged @heddn about yesterday!

See #3108302-64: [PP-2] Field formatter & widget settings: fall back to default if missing plugin → for a core patch that has not needed to change since 2021, but now does, if we want tests to pass on PHP >=8.2.

I wrote ~500 LoC of new test coverage to figure out in which heuristic was clustering the d7_taxonomy_vocabulary_translation:* plugins into the appropriate clusters, because that is the last merge-blocking test failure preventing this from getting wrapped up.

That led me to getting the necessary information to continue debugging:

-    'd7_field_instance:comment:a_thirty_two_character_type_name' => Array &112 (
+    'd7_taxonomy_vocabulary_translation:vocabfixed' => Array &112 (
+        'heuristic' => 'entity_depending_config'
+        'cluster' => 'LIFTED-VocabFixed taxonomy terms'
+    )
+    'd7_taxonomy_vocabulary_translation:vocablocalized' => Array &113 (
+        'heuristic' => 'entity_depending_config'
+        'cluster' => 'LIFTED-VocabLocalized taxonomy terms'
+    )
+    'd7_taxonomy_vocabulary_translation:vocabtranslate' => Array &114 (
+        'heuristic' => 'entity_depending_config'
+        'cluster' => 'LIFTED-VocabTranslate taxonomy terms'
+    )

— https://git.drupalcode.org/project/acquia_migrate/-/jobs/649770

Also, the research I did ~1 year ago to get AM:A running on Drupal 10 has proven invaluable, because the last failure in DrushCommandsTest was very hard to debug … and #3260208-20: FilterPermissions' use of URLs can cause module installations to fail → turns out to still contain the solution.

Updating the recommendation to apply that core patch too results in that test passing at last 👍

WRT #19:

1. https://git.drupalcode.org/project/acquia_migrate/-/jobs/649876 → now it shows an explicit failure specifically when D9 clusters things differently than D10, so the next time this happens, it'll be comparatively trivial to spot instead of spending >1 day chasing this!
2. Root cause found! #2953111: Only migrate role permissions that exist on the destination → caused this change in behavior, because it made d7_user_role depend on d7_taxonomy_vocabulary_translation , which caused the ContentEntityDependingConfig heuristic to no longer match! 😬

https://git.drupalcode.org/project/acquia_migrate/-/tree/recommendations-10 is now live!

Time to merge, because:

The only compromise: Nightwatch tests are not yet passing on Drupal 10 … on GitLab CI. They do pass fine locally. For a mysterious reason, they're not being found on GitLab CI but are found locally. Blocking the Drupal 10 support on that is unreasonable, especially because the tests do still run fine on Drupal 9 on GitLab CI 👍

Merged!

I removed the hacks that were necessary in the CI pipeline prior to merging, and the resulting 1.9.x branch's pipeline is working fine 👍

Automatically closed - issue fixed for 2 weeks with no activity.

jienckebd → closed merge request !40

Hey Wim! I use the D7 Biblio module, which had a recommended migration for AMA:D9. But when this was merged, the migration was removed. I don't understand why, given that the replacement module (bibcite) has had a recommended (though not stable) release since last year.

Do you know why it was removed?