D10: Run crawler script to update generated.json

I am a little confused by the output in statistics

specifically this part:

  -  (of which ~ 388 generated recommendations overridden by curated recommendations)
  + (of which ~ 311 generated recommendations overridden by curated recommendations)

Did we lose 77 curated recommendations in this process somehow?

❌ pipelines failing!

Array
(
    [package] => drupal/auto_entitylabel
    [constraint] => ^3
    [replaces] => Array
        (
            [name] => auto_nodetitle
        )
    [vetted] => 
)
conflicts with
Array
(
    [package] => drupal/auto_entitylabel
    [constraint] => ^3.0
    [replaces] => Array
        (
            [name] => auto_entitylabel
        )
    [vetted] => 
)

I think it is because ^3 vs ^3.0. Seeing what happens if I change that in curated.json

I think there are probably quite a few of these, another failure:

(
    [package] => drupal/chosen
    [constraint] => ^4
    [replaces] => Array
        (
            [name] => chosen_ajax
        )
    [vetted] => 
)
conflicts with
Array
(
    [package] => drupal/chosen
    [constraint] => ^4.0
    [replaces] => Array
        (
            [name] => chosen
        )
    [vetted] => 
)

I dont want to spam pipelines so maybe I can try to identify which ones differ and fix them all in one go .

I figured out i could run

php scripts/test-unvetted-modules-composer-installability.sh recommendations.json /tmp/test-general-installability-unvetted 10.2.3

to test this locally (to avoid the aforementioned spamming of pipelines 😄). However...i think this is exposing a larger issue. Some of the generated recommendations, for example:

        {
            "package": "drupal/node_revision_delete",
            "constraint": "^2.0",
            "replaces": {
                "name": "node_revision_delete"
            },
            "vetted": false
        },

conflict heavily with the curated.json value, which is:

            "package": "drupal/node_revision_delete",
            "constraint": "^1",
            "install": [
                "node_revision_delete"
            ],
            "replaces": {
                "name": "node_revision_restrict"
            },
            "vetted": false
        },

even though it is vetted:false in curated.json it still feels like something else needs to happen here. Another example is rabbit hole:

Array
(
    [package] => drupal/rabbit_hole
    [constraint] => ^1
    [replaces] => Array
        (
            [name] => rh_file
        )

    [install] => Array
        (
            [0] => rh_file
        )

    [vetted] =>
)
conflicts with
Array
(
    [package] => drupal/rabbit_hole
    [constraint] => ^2.0
    [replaces] => Array
        (
            [name] => rabbit_hole
        )

    [vetted] =>
)

I'm not sure i feel comfortable updating these in curated.json without actually testing them myself locally, but I guess if its not marked as "vetted" that is ok? Updated the MR with updates to the failing packages and an added conditional in the test-unvetted-modules-composer-installability.sh script be more lenient with version checks to start 😊.

I thought the constraints in curated.json would have superseded those in generated.json, but this appears to be throwing an error. My changes have also broken things quite substantially in pipelines 😬 sooo im rethinking this at the moment

Here are the current issues:

  Problem 1
    - Root composer.json requires drupal/ace_editor ^2.0 -> satisfiable by drupal/ace_editor[2.0.0-alpha1, 2.0.0-beta1, 2.0.0-beta2, 2.0.x-dev].
    - drupal/ace_editor[2.0.0-alpha1, ..., 2.0.x-dev] require npm-asset/ace-builds ~1.0 -> could not be found in any version, there may be a typo in the package name.

This one will get addressed with https://www.drupal.org/project/acquia_migrate/issues/3417209 📌 Obsolete vetted modules to review - 1/10 Active .

  Problem 2
    - Root composer.json requires drupal/agls ^1.2 -> satisfiable by drupal/agls[1.2.0, 1.x-dev].
    - drupal/agls[1.2.0, ..., 1.x-dev] require drupal/metatag ^1.0 -> found drupal/metatag[dev-1.x, 1.0.0-beta1, ..., 1.x-dev (alias of dev-1.x)] but it conflicts with your root composer.json require (^2).

I would argue that the recommendation for metatag @ version ^2 supersedes drupal/agls (based on installation usage). So drupal/agls should be moved to curated.json with a note stating that it needs to be manually installed due to the need to downgrade metatag.

  Problem 3
    - Root composer.json requires drupal/codemirror_field ^2.0 -> satisfiable by drupal/codemirror_field[2.0.1].
    - drupal/codemirror_field 2.0.1 requires codemirror/codemirror5 ^5 -> could not be found in any version, there may be a typo in the package name.

This module requires manual installation due to needing to alter the repositories section of composer.json by hand. Should update this in curated.json

  Problem 4
    - Root composer.json requires drupal/commerce_realex ^3.0 -> satisfiable by drupal/commerce_realex[3.0.0, 3.0.x-dev].
    - drupal/commerce_realex[3.0.0, ..., 3.0.x-dev] require annertech/rxp-js 1.3.1.21 -> could not be found in any version, there may be a typo in the package name.

The recommendation from the previous version of recommendations.json still applies to this module.

  Problem 5
    - Root composer.json requires drupal/contextly ^3.0 -> satisfiable by drupal/contextly[3.0.0].
    - drupal/contextly 3.0.0 requires contextly/contextly-kit 5.0.10 -> could not be found in any version, there may be a typo in the package name.

This recommendation still applies to the current module.

  Problem 6
    - Root composer.json requires drupal/feeds_ftp_fetcher ^2.0 -> satisfiable by drupal/feeds_ftp_fetcher[2.0.0-alpha1, 2.x-dev].
    - drupal/feeds_ftp_fetcher[2.0.0-alpha1, ..., 2.x-dev] require ext-ftp * -> it is missing from your system. Install or enable PHP's ftp extension.

This seems to be more an issue with the CI - i think we need to enable a new extension for this to install. Locally I am able to install it.

  Problem 7
    - Root composer.json requires drupal/lightgallery ^1.4 -> satisfiable by drupal/lightgallery[1.4.0, 1.x-dev].
    - drupal/lightgallery[1.4.0, ..., 1.x-dev] require sachinchoolur/lightgallery ^1.2.21 -> could not be found in any version, there may be a typo in the package name.

The same recommendation apples from the previous version.

  Problem 8
    - Root composer.json requires drupal/monster_menus ^9.1 -> satisfiable by drupal/monster_menus[9.1.0-rc1, 9.1.0-rc2, 9.1.0-rc3, 9.1.0].
    - drupal/monster_menus[9.1.0-rc1, ..., 9.1.0] require drupal/core ^9.4 || 10 -> found drupal/core[9.4.0-alpha1, ..., 9.5.x-dev, 10.0.0] but it conflicts with your root composer.json require (10.2.3).

This module might be ok to use (haven't tested) but cannot be installed due to a typo in its composer.json file, here. Based on the module .info file this should be compatible with ^10 which would install on 10.2.3, but this requirement in composer.json prevents installation.

  Problem 9
    - Root composer.json requires drupal/signaturefield ^1.1 -> satisfiable by drupal/signaturefield[1.1.0, 1.x-dev].
    - drupal/signaturefield[1.1.0, ..., 1.x-dev] require npm-asset/signature_pad ^2.3 -> could not be found in any version, there may be a typo in the package name.

The same recommendation applies.

  Problem 10
    - Root composer.json requires drupal/smartmenus ^2.0 -> satisfiable by drupal/smartmenus[2.0.1-beta1, 2.0.1-beta2, 2.0.x-dev].
    - drupal/smartmenus[2.0.1-beta1, ..., 2.0.x-dev] require drmonty/smartmenus 1.1.1 -> found drmonty/smartmenus[dev-master, v0.9.6, v0.9.7, 1.0.0, 1.0.1, 1.1.0] but it does not match the constraint.

This module requires manual installation. See here.

  Problem 11
    - Root composer.json requires drupal/swagger_ui_formatter ^4.0 -> satisfiable by drupal/swagger_ui_formatter[4.0.0].
    - drupal/swagger_ui_formatter 4.0.0 requires php ~8.1.0 -> your php version (8.2.15) does not satisfy that requirement.

This module is not yet compatible with PHP 8.2. It also appears the issue queue is disabled → ? Odd.

  Problem 12
    - drupal/drupal[8.4.0-alpha1, ..., 8.7.x-dev] require composer/installers ^1.0.24 -> found composer/installers[v1.0.24, ..., 1.x-dev] but it conflicts with your root composer.json require (^2.0).
    - drupal/at_tools[3.6.0, ..., 3.x-dev] require drupal/adaptivetheme ^4.0 -> satisfiable by drupal/adaptivetheme[4.0.0, 4.1.0, 4.x-dev].
    - drupal/adaptivetheme[4.0.0, ..., 4.x-dev] require drupal/core ^8.8 || ^9 -> satisfiable by drupal/drupal[8.4.8, 8.5.15, 8.6.18, 8.7.14].
    - Root composer.json requires drupal/at_tools ^3.6 -> satisfiable by drupal/at_tools[3.6.0, 3.x-dev].

I am a little confused on this one. The project composer.json file says it needs drupal/adaptivetheme: ^8.4 but i dont see that as a realease/tag. → . Maybe this tracks to 8.x-4.x? If so, then the issue is that 4.x of drupal/adaptivetheme is incompatible with Drupal 10. → . It also appears that at_tools has been abandoned and now the work is being done in at_tool →

I made this changes locally and then saw this error:

  Problem 1
    - drupal/drupal[8.4.0-alpha1, ..., 8.7.x-dev] require composer/installers ^1.0.24 -> found composer/installers[v1.0.24, ..., 1.x-dev] but it conflicts with your root composer.json require (^2.0).
    - drupal/basic_cart 7.0.0 requires drupal/easy_install * -> satisfiable by drupal/easy_install[dev-1.x, dev-3.x, dev-4.x, dev-5.x, dev-6.x, dev-7.x, dev-8.x, dev-9.x, dev-10.x, 1.0.0, 1.x-dev, 2.0.0-alpha1, 3.x-dev, 4.0.0-alpha2, 4.x-dev, 5.0.0-alpha3, 5.x-dev, 6.0.0-alpha4, 6.x-dev, 7.0.0-alpha5, 7.x-dev, 8.0.0-alpha6, 8.x-dev, 9.0.0-rc1, 9.x-dev, 10.0.0, ..., 10.x-dev].
    - drupal/easy_install[dev-10.x, 10.3.0, ..., 10.x-dev] require drupal/core ^8 || ^9 -> satisfiable by drupal/drupal[8.4.0-alpha1, ..., 8.7.x-dev].
    - drupal/easy_install[dev-1.x, dev-3.x, dev-4.x, dev-5.x, dev-6.x, dev-7.x, dev-8.x, dev-9.x, 1.0.0, ..., 1.x-dev, 2.0.0-alpha1, 3.x-dev, 4.0.0-alpha2, ..., 4.x-dev, 5.0.0-alpha3, ..., 5.x-dev, 6.0.0-alpha4, ..., 6.x-dev, 7.0.0-alpha5, ..., 7.x-dev, 8.0.0-alpha6, ..., 8.x-dev, 9.0.0-rc1, ..., 9.x-dev, 10.0.0, ..., 10.2.0] require drupal/core ~8.0 -> satisfiable by drupal/drupal[8.4.0-alpha1, ..., 8.7.x-dev].
    - Root composer.json requires drupal/basic_cart ^7.0 -> satisfiable by drupal/basic_cart[7.0.0].

This module does not appear to be installable right now due to an issue with Easy Install being incompatible with Drupal 10. Easy install needs to be patched - see https://www.drupal.org/project/easy_install/issues/3297022 📌 Automated Drupal 10 compatibility fixes Needs review .

Then I hit:

  Problem 1
    - drupal/block_country dev-1.x requires drupal/ip2country ^1.9 -> found drupal/ip2country[dev-1.x, 1.9.0, 1.10.0, 1.11.0, 1.x-dev (alias of dev-1.x)] but it conflicts with your root composer.json require (^2).
    - drupal/block_country 1.x-dev is an alias of drupal/block_country dev-1.x and thus requires it to be installed too.
    - Root composer.json requires drupal/block_country 1.x-dev -> satisfiable by drupal/block_country[1.x-dev (alias of dev-1.x)].

This module does not have a stable release. Stopping for now...

dan612 → changed the visibility of the branch 3420498-update-generated to hidden.

Currently we have 171 generated recommendations. After this there are 1423 generated recommendations.

I realized this morning that what I was doing in my previous efforts was just re-vetting all those modules which failed to install cleanly...which is what we want to do anyway but there are probably higher priorities 😄. that being said, i think i should break some of my comments in #7 📌 D10: Run crawler script to update generated.json Active into tickets as the work (or groundwork) is already done now.

New idea. What if we just omit all the d10-install-breaking-modules from generated.json? This would mean we still get an additional ~1,200 modules which install successfully on D10 and it would unblock this issue, which relies on basically vetting every module which fails to cleanly install via composer.

Locally I reset things back to recommendations-10, re-ran make -B and cofirmed the mass update in generated.json. At this point my statistics showed:

$ cat ./statistics.txt                                                                       
410 recommendations (curated)
  (of which 364 contrib recommendations)
1480 recommendations (generated)
1790 recommendations (total)
  (of which 89 vetted recommendations)
    (destination: targeting 73 composer packages)
    (destination: targeting 121 Drupal modules)
    (source: migrating from 231 Drupal modules)
    (source: obsoleting 63 Drupal modules)
  (of which 63 obsolete recommendations)
  (of which 3 patched recommendations)
  (of which ~ 310 generated recommendations overridden by curated recommendations)
41 unique patches, or 13.7 (avg) per patched recommendation for a total of 41 (36 max)
  (of which 3 patches are not migration-specific, but necessary bugfixes)

Then i added the modules i knew from previous efforts to be breaking to a new file, iterated over it and removed those from generated.json. Then re-ran the CI process locally to make sure things worked. Whenever there was a failure, i removed it from generated.json and tried again. I attached the list of packages which caused failure → . The result appears to be a passing CI pipeline now 🎉. I will leave it up for discussion what other implications this has and if it is a good idea or not 😬

I believe it is an improvement 🤞 because the majority of these modules should work on Drupal 10 (considering they have a D10 compatible release tagged). So even though we haven't tested them or vetted migration paths, i believe it makes things easier for the end user because they will come preinstalled (but not enabled) if they had them on Drupal 7.

#3: Hm … that is odd. 🤔 I think it might be related to #4 — I forget how exactly curated.json and generated.json are merged into recommendations.json, I bet it's related to that. Otherwise: a bug in scripts/create-statistics.sh is not impossible either 😅 This evolved on an as-needed and "good enough" basis.

#6: Yes, whenever these kinds of conflicts arise, we are forced to override the generated recommendation in curated.json, and just keep the vetted: false. In this case, a recommendation is generated for the D7 node_revision_delete module (great!), but it recommends a different D10 package version than the curated recommendation for node_revision_restrict. And that's a problem, because we cannot install multiple package versions. Usually, the solution is to just add more overrides in curated.json. But in this case, I bet I made a mistake while trying to determine the minimal viable subset of the D9 curated.json that would work for D10.

If you're wondering why the node_revision_restrict non-vetted curated recommendation even exists, it's because that's a D7 submodule of the node_revision_delete module. The script that generates generated.json can only know about projects == "main module names", not any submodules. So any submodule that we've encountered along the way while working on the D9 recommendations, we've added a matching non-vetted curated recommendation for, to provide maximal coverage for AM:A users (otherwise it'd seem like there is no equivalent!).

I'm not sure i feel comfortable updating these in curated.json without actually testing them myself locally, […]

No need for testing locally — thanks to vetted: false 👍 These modules won't be Drupal-installed, only composer-installed.

I thought the constraints in curated.json would have superseded those in generated.json, but this appears to be throwing an error

They do! But it's "keyed" by replaces.name aka the D7 module they replace. And hence that automatic overriding does NOT work for submodules. See https://git.drupalcode.org/project/acquia_migrate/-/blob/recommendations....

I think 👆 should help you get everything that the script puts in generated.json to a passing state. But … I'm pragmatic 😁, and I agree with you in #10 that the current MR is already a big net improvement, so merged that! Keeping this open for a second MR on this issue now that I've explained why those conflicts were occurring (submodules!), which should make it trivial for you to resolve the conflicts 🤓

The changes you made in curated.json make sense — they're all intended to align with the generated.json's version numbers for the non-submodule package versions, for example the D7 flood_unblock had ^2 in curated.json, but generated.json contains:

+        {
+            "package": "drupal/flood_control",
+            "constraint": "^2.3",
+            "replaces": {
+                "name": "flood_control"
+            },
+            "vetted": false
+        },
+        {
+            "package": "drupal/flood_control",
+            "constraint": "^2.3",
+            "install": [
+                "flood_control"
+            ],
+            "replaces": {
+                "name": "flood_unblock"
+            },
+            "vetted": false
+        },

… and so to avoid the need to repeat both of those recommendations in curated.json to set them to ^2 both, just changing the submodule's recommendation in curated.json to ^2.3 is less work/effort 👍

@dan612 did you see #12?

😁 i did! however I am not sure we are talking about the same issue. I encountered 2 types of issues when working on this:

1. Issues where version constraints differed between curated.json and generated.json -- as you described in your previous comment, these were mostly submodules and easy to fix by manually changing the version.
2. Modules which get picked up by generated.json but are not installable via composer

The second issue above is where this is stuck, none of these modules - https://www.drupal.org/files/issues/2024-02-10/d10-composer-install-brea... → - were able to be added to a D10 project via composer, cleanly. For example drupal/field_properties:

Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Root composer.json requires drupal/field_properties ^1.0@alpha -> satisfiable by drupal/field_properties[1.0.0-alpha2].
    - drupal/field_properties 1.0.0-alpha2 requires drupal/core ^8 -> found drupal/core[8.0.0-beta6, ..., 8.9.x-dev] but the package is fixed to 10.2.3 (lock file version) by a partial update and that version does not match. Make sure you list it as an argument for the update command.

or smartmenus:

[6:53:43] ~/Sites/d7-upgrades/the18-d10 [main] $ lando composer require 'drupal/smartmenus:^2.0@beta'
./composer.json has been updated
Running composer update drupal/smartmenus
Gathering patches for root package.
Loading composer repositories with package information
Updating dependencies
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Root composer.json requires drupal/smartmenus ^2.0@beta -> satisfiable by drupal/smartmenus[2.0.1-beta1, 2.0.1-beta2].
    - drupal/smartmenus[2.0.1-beta1, ..., 2.0.1-beta2] require drmonty/smartmenus 1.1.1 -> found drmonty/smartmenus[dev-master, v0.9.6, v0.9.7, 1.0.0, 1.0.1, 1.1.0] but it does not match the constraint.

So for these...i feel like we would have to vet each on against D10 and provide a curated recommendation in order for this to be complete.

To be honest a lot of these packages look as though they are in alpha/beta and dont have a stable release to begin with...which makes me wonder why the crawler script was even picking them up because it should only get D10 packages when:

 * For the given project, try to derive a D7-D10 migration recommendation.
 *
 * Projects are ignored if:
 *
 * - Their owner has declared that it has been replaced.
 * - Their status is anything other than receiving maintenance fixes only or
 *   under active development (e.g. not obsolete).
 * - They do not have a Drupal 7 release.
 * - They do not have a Drupal 10 release.

the above snippet was taken from crawl.php

Ahhh!

Looks like I mixed things up indeed — sorry about that 🙈

Modules that get automatically added to generated.json but are not composer-installable are worth adding to curated.json explicitly, because … then we can inform the user that right now, there's no installable release, and that there's a particular issue where that's being worked on. Now for the cool part: linking to such an issue will cause scripts/dgo_issue_tracker.sh to get updated automatically when you run make, meaning we'll get notified through our CI when that issue lands upstream and our "nope doesn't work" recommendation in curated.json has become stale! 😊

To be honest a lot of these packages look as though they are in alpha/beta and dont have a stable release to begin with...which makes me wonder why the crawler script was even picking them up because it should only get D10 packages when:

Well, that module has a stable security-covered D7 release, so any D7 site using it and wanting to move to D10 will want to know if there's any release available that might help avoid writing new code/doing a custom migration, because then there's at least some release to test. So https://www.drupal.org/project/field_properties → falls under that category. You'll find plenty of -dev constraints in the D9 recommendations too: https://git.drupalcode.org/project/acquia_migrate/-/blob/recommendations...

made an MR to show i am still noodling on this 😬

I re-ran the make -B command, then my process was as follows:

1. Run sh ./scripts/test-unvetted-modules-composer-installability.sh recommendation.json /tmp/test-general-installability-unvetted 10.2.3
2. Go to new directory and run composer install --dry-run
3. Move each failure to curated.json with a note as to why it fails to install, and vetted: false
4. Run composer remove <package> then re-run composer install --dry-run

Basically just worked through all the various failures during composer installation, went to the projects to review why it failed and then updated the note with the findings. Then my pipelines jobs started to fail due to PHP 8.2 incompatibility - so I made a few more updates to address that after changing versions and testing again locally

Epic work!

My #1 concern is that many of these will go outdated without us knowing. Most of these can and should have d.o issue links. The majority of them do, but I also saw many that still need it.

Once you add that, I'll instantly merge this!

P.S.: the TabT module there is a BLAST FROM THE PAST! 🤣🏓🏓🏓🏓🏓

I think i covered all issues, but LMK if i missed something 😎

I also think that I found a bug in the crawler that was stopping crawling ~400 pages in, when there are over 650 pages of results to iterate through. Probably best to keep that in the original parent issue and break out another update once possible...assuming this one is OK.

dan612 → closed merge request !31

Probably best to keep that in the original parent issue and break out another update once possible...assuming this one is OK.

By "original parent issue" you mean 📌 Refine the crawler that generates `generated.json` to support full Composer constraint expressions Active ?

I really wanted to RTBC + merge this. But I don't understand 5 of the overrides in curated.json. Could you add a note to them to help me understand (and help you evolve it in the future)? 🙏

Made some updates 🥹 - we also had a pipelines failure due to an issue getting resolved - https://www.drupal.org/project/monster_menus/issues/3422059 🐛 Cannot install module on any version of Drupal 10 other than 10.0.0 Fixed . So the system is working! 🎉

but since it might change the functions of the crawler i thought perhaps better off in the parent.

👍

No more remarks on the MR now either — you were able to remove ~50% of the ones I was unsure about, and the ones that remain now have a clear note 🥳

Automatically closed - issue fixed for 2 weeks with no activity.