Upgrading Drupal core to a new major version is very difficult when Drush is also installed

Thanks @joachim, adding another related issue.

I don't understand why you actively refuse to follow both the documentation and the suggestions on Slack. Which is to run composer require --no-update followed by a composer update.

composer require without --no-update means it runs a composer update internally. Instead of the --no-update, you can also edit composer.json manually, that's all it does.

-W means tells composer to update a package and its dependencies, but no more.

The problem with drush and drupal/core is the same it's been also for Drupal 8/9, both depend on some symfony components, but not the same. These symfony components can't be installed in different major versions, so they need to be updated, but with -W, you don't allow to update the Symfony components that drush depends on.

Yes, the composer error you get is weird, one way to get a better output is to not use drupal/core-recommended as explained in the link I sent you (https://md-systems.notion.site/Major-Drupal-upgrades-with-composer-68c8c...):

composer remove --no-update drupal/core-recommended
composer require --no-update drupal/core
composer require -W drupal/core:^10
  Problem 1
    - drupal/core[10.0.0, ..., 10.0.11] require symfony/console ^6.2 -> found symfony/console[v6.2.0, ..., v6.3.4] but these were not loaded, likely because it conflicts with another require.
    - drupal/core[10.1.0, ..., 10.1.5] require symfony/console ^6.3 -> found symfony/console[v6.3.0, v6.3.2, v6.3.4] but these were not loaded, likely because it conflicts with another require.
    - Root composer.json requires drupal/core ^10 -> satisfiable by drupal/core[10.0.0, ..., 10.1.5].

Still not great, but closer to the source, there's conflicts related to symfony/console.

Whether or not you are using drupal/core or drupal/core-recommended, what works for this case is including drush/drush in the composer require command, because then you allow composer to update those dependencies as well, and then that works:

composer require -W drupal/core-recommended:^10 drush/drush

Because what you are doing is giving composer an impossible task, and it gives you a confusing error. The best you can hope for is getting a better error.

Or you could follow the documentation and do a composer update, to allow it to update all packages without restrictions (other than those set in composer.json of course).

There _are_ challenging issues with composer updates when using many modules, and I had a case as documented on the link above where I had to give composer a nudge in the right direction even with a full composer update, but this is not one of them.

So I agree that it is easy to not find that page, and then just try composer require drupal/core-recommended:^10 and then this error message from composer is confusing:

composer require -W drupal/core-recommended:^10
./composer.json has been updated
Running composer update drupal/core-recommended --with-all-dependencies
Loading composer repositories with package information
Updating dependencies
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - drupal/core-recommended[10.0.0, ..., 10.1.5] require psr/log ~3.0.0 -> found psr/log[3.0.0] but these were not loaded, likely because it conflicts with another require.
    - Root composer.json requires drupal/core-recommended ^10 -> satisfiable by drupal/core-recommended[10.0.0, ..., 10.1.5].


Installation failed, reverting ./composer.json and ./composer.lock to their original content.

Berdir's alternative to the docs page to do

composer require -W drupal/core-recommended:^10
./composer.json has been updated
Running composer update drupal/core-recommended --with-all-dependencies
Loading composer repositories with package information
Updating dependencies
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - drupal/core-recommended[10.0.0, ..., 10.1.5] require psr/log ~3.0.0 -> found psr/log[3.0.0] but these were not loaded, likely because it conflicts with another require.
    - Root composer.json requires drupal/core-recommended ^10 -> satisfiable by drupal/core-recommended[10.0.0, ..., 10.1.5].


Installation failed, reverting ./composer.json and ./composer.lock to their original content.

composer require drupal/core-recommended:^10.1 drush/drush works, although of course with more dependencies, that list could get longer.

Thoughts:

1. Drush specifies a conflict with older drupal/core versions:

 "conflict": {
    "drupal/core": "< 10.0",
    "drupal/migrate_run": "*",
    "drupal/migrate_tools": "<= 5"
  },

What if it specified a confict with < 10.0 || >= 11.0 would that mean composer would mention drush in the error message instead of/as well as psr/log?

Drush itself is not the problem, this works too:

composer require -W drupal/core-recommended:^10 drush/drush:^11

I only included drush/drush there for it's dependencies (combined with the -W), the drush update itself is optional.

But either way, both of those commands are just to demonstrate the problem. A major core update should IMHO *always* use composer update without arguments, after updating composer.json either manually or with --no-update. There are so many intermediate packages and indirect dependencies between them that it's hard to resolve otherwise if you had 50+ contrib modules to the mix.

> So I agree that it is easy to not find that page, and then just try composer require drupal/core-recommended:^10 and then this error message from composer is confusing:

Yeah, that seems like something that we can work on. Googling for drupal 10 upgrade finds a number of different pages:

https://www.drupal.org/docs/upgrading-drupal/upgrading-from-drupal-8-or-... →
https://www.drupal.org/docs/upgrading-drupal/upgrading-from-drupal-8-or-... →

And then on that page is a linked "Update" that took me a while to find too, that points to https://www.drupal.org/docs/updating-drupal → , then you go to the composer page and https://www.drupal.org/docs/updating-drupal/updating-drupal-core-via-com... → , which then points to https://www.drupal.org/docs/upgrading-drupal/drupal-8-and-higher → , which is on the same level as the one this journey started, but it is not linked or referenced there.

I think composer when it does Your requirements could not be resolved to an installable set of packages. and then when there is just one Problem 1 .... require psr/log ~3.0.0 could suggest running composer why-not psr/log:^3.0 - that seems like it ought to be doable

Something like:

Installation failed, reverting ./composer.json and ./composer.lock to their original content. Try composer why-not psr/log: ^3.0 to find out why.. Although in this case, composer why-not doesn't mention drush, you'd have to recursively run why-not on symfony/console or whatever you got from the first why-not.

Or if composer can't/won't do it, maybe we can add a plugin that does something?

A major core update should IMHO *always* use composer update without arguments, after updating composer.json either manually or with --no-update. There are so many intermediate packages and indirect dependencies between them that it's hard to resolve otherwise if you had 50+ contrib modules to the mix.

This might be the case but it feels like a problem that's more-or-less specific to Drupal and therefore not easy to discover that you need to do it.

Also, update and upgrade are not the same. In the wiki update is used for moving from one minor version to another minor version and upgrade for moving from one major version to another major version. So, following 'update' links is not likely to get you where you want to be if you are upgrading. The upgrade → and the update → docs are in different guides. Yes, I am aware that in conversation people will interchange them.

If you go to Upgrading from Drupal 9 to Drupal 10 → then select Overview → you find the composer instructions linked on that page in the Upgrade → section.

The page → 'not linked to anywhere' mentioned in #5 is for Upgrading from Drupal 8 to Drupal 9 so may not be suitable for 9 -> 10.

I have just upgraded two very large and complex sites and ran into the exact same error message as joachim. I have not found drush to be a special case. The error message as shown in the issue summary is not helpful but if I remove composer.lock it begins to show -- one at a time but still -- what the offending project is. I think but of course do not know this is a composer bug: why does the behavior differ between -W and removed composer.lock? Isn't -W supposed to be the same?

Once I got useful error messages, it turned out the drupal.org recommended way of installing projects with composer require 'drupal/better_exposed_filters:^5.2' makes composer unable to install better_exposed_filters 6.0. There's no "change my root requirements constraints to their latest versions" composer command. Of course this could introduce breakage but that's no reason to leave the user in a lurch. I have fixed this with a script (originally from stackoverflow) which uses the output of composer outdated --direct > outdated.txt to do this edit. This is a definitely a missing composer feature. (Footnote: of course the manual directs you to use upgrade status and upgrade your contrib first. This is not feasible manually. https://i.imgur.com/zoF8iwz.png not all upgrades necessary are branch but too many are. And of course these are only the drupal projects, you might have other php projects which depend on Symfony which we bump so they need to be bumped too. So this can't be solved on the drupal side, this is a composer issue.)

- why does the behavior differ between -W and removed composer.lock? Isn't -W supposed to be the same?

The way I understand it in the specific case of the issue summary is that -W will update all of Drupal's dependencies.

However, drush/drush doesn't depend on Drupal at all, it just happens to depend on some of Drupal's dependencies, so it's not included in -W. This is why #5 makes a difference by also allowing drush and its dependencies to update.

Also, update and upgrade are not the same. In the wiki update is used for moving from one minor version to another minor version and upgrade for moving from one major version to another major version

I'm not sure there is a hard distinction any more. This split comes from the Drupal 6 to 7 era (or before) when an upgrade definitely meant that custom code and themes had to be ported. Upgrading a major version is still harder than a minor but it's possible on a smaller or extremely pre-prepared site where everything is compatible with 9/10 that the only difference is the composer commands. The pre-preparation is of course still a huge difference between minors and majors so I'm undermining my point as I type this, but it's for completely different reasons than earlier Drupal core versions. When we have Drupal 10 supported longer into Drupal 11's existence it's more likely that sites will be Drupal 11-ready by default if they're up to date with contrib within a year or 18 months and then it really will just be the composer differences for core (for a subsection of sites).

Although I have never been able to find documented evidence I do recall migration discussions, probably about the one click method, about update vs upgrade and we made a deliberate chose to use upgrade for the UI. And that has stayed with me. And the distinction between upgrade and update is not specific to Drupal. There are links in this comment #2709507-22: [META] Remove outdated references to "upgrade" from core → that show that other projects make the same distinction.

Phew, where to start :)

Update vs upgrade:
I'm also conflicted, I agree with @catch that there is no longer as clear a distinction, but there are still some differences. Symfony also uses upgrade: https://symfony.com/doc/current/setup/upgrade_major.html. I think it's important that we always provide more context than just update or upgrade, which we mostly do, just like Symfony, which states "Upgrade to the next major version". And our Update intro does the same on https://www.drupal.org/docs/updating-drupal/options-for-updating-drupal-... → . Personally I think that distiction could be on the parent page already because I suspect a lot of people will skip the intro page and directly go to composer because that's what they use, but that's pretty minor thing. And I see that @quietone already added the link to composer upgrade, so my main complaint is already resolved. Great!

On composer require -W, composer update and removing lock file:
There's clearly still a lot of confusion about what each of these commands with their different options do. Comparing again with the Symfony docs, I really like their approach. They ignore composer require completely and instead suggest to edit the composer.json file. They have a lot more variations between projects, each project uses different symfony components and a lot more than we have drupal/core parts, but even we do have some variations, some use drupal/core, others recommended, we don't use the project message for example. Showing that with a diff IMHO makes it much clearer what is happening than some complex magic require commands that might or might not apply to your project. We could still have a line below that says instead of editing the file, you can also use composer require --no-update drupal/core-something. They do then provide 3 options with decreasing strictness. First only their components, then with dependencies, then everything. But their dependency chains are less complex, so IMHO, we should just directly jump to composer update (as we do now, maybe with a bit more explanation).

why does the behavior differ between -W and removed composer.lock? Isn't -W supposed to be the same?

The way I understand it in the specific case of the issue summary is that -W will update all of Drupal's dependencies.

However, drush/drush doesn't depend on Drupal at all, it just happens to depend on some of Drupal's dependencies, so it's not included in -W. This is why #5 makes a difference by also allowing drush and its dependencies to update.

Yes. composer require -W package or composer update -W package is definitely *not* the same thing as removing the lock file. composer update -W drupal/core updates just drupal/core and its dependencies, not everything. Note that whether or not drush/drush depends on drupal/core doesn't make a difference in this case, drupal/core would need to depend on drush/drush.

To give one specific, simplified example (it's not actually happening like this in case, see below), drush/drush 11 depends on "symfony/filesystem ^4.4 || ^5.4 || ^6.1". Drupal 9 does not depend on symfony/filesystem. It however depends on other symfony packages version 4 only, lets assume* that composer then also installs symfony/filesystem v4. Now you update to drupal/core to 10.1. drupal/core 10.1 still doesn't depend on symfony/filesystem. But due to dependencies and conflicts between symfony packages, composer would need to update symfony/filesystem to version 6 as well. But with composer update -W drupal/core, it's not allowed to do so as it's not one of the dependencies of drupal/core.

* Again, this is not what actually happens in this minimal example, if you require drush/drush, it will install symfony/filesystem v6 because there is no conflict between the symfony components v4 that Drupal uses and symfony/filesystem. But there could be with other modules and there are other not shared dependencies that cause this exact problem.

Now, what I would expect to happen is that composer update without any arguments and removing the composer.lock file should behave in the same way. But I'm not 100% sure, there might be edge cases where that's not the case.

I moved beyond simple scripts to composer plugins.

1. chx/jump upgrades every direct dependency in composer.json to their latest version with a restriction in the major version only
2. chx/drupal-issue-fork provides composer drupal-issue-fork which can be used to set up an issue fork as explained on the Core version compatibility fixes for modules with unmerged changes → handbook page.

using composer jump first makes upgrade a breeze in my experience: composer jump, rm composer.lock, composer install. If something fails, use the composer drupal-issue-fork command to use a forked branch instead of the last release and run install again until satisfied.

Hi,

I was able to fix this with:
$ composer require -W drupal/core-recommended:^10.0.0 drush/drush:^12.5 guzzlehttp/guzzle twig/twig symfony/yaml:^6.0 drupal/core:^10 drupal/seven --ignore-platform-reqs

I was getting error of drupal/console version dependency. after fix this above command work to me and updated core with drupal10.2

#4. Thank you!