Update from Yarn 1 to Yarn 4

Yarn 4 in rc. So we may test it in 11.x

Seems updating caused some test errors.

Looks like nightwatch needs update

The IS needs a (hopefully minor) update to reflect the current status of affairs. Yarn 4 is now fully released.

It can be reviewed and tested manually because pipeline failure related to old version of yarn used in gitlab CI
#49

Do we really have to ship the code for yarn inside the core directory? Can we instead rely on the user installing yarn themselves?

We also need to make the GitLab CI scripts (which also live in the repo) use the new version.

Scripts are in repo but Docker templates outside.
https://git.drupalcode.org/project/drupalci_environments/-/blob/dev/dock...

I don't think we need to run extra scripts in CI container in this ticket.

Regarding the pipeline failure: I ran into this same issue in trying to use Yarn 3.x and 4.x for the current 2.x revamp of the RefreshLess module → (see 📌 Hotwire Turbo minimum viable implementation Active if you're interested by the way); here's our current .gitlab-ci.yml that took me a lot of swearing and late hours last night to finally get working; it's well documented so should explain everything.

The Needs Review Queue Bot → tested this issue. It fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

This does not mean that the patch needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide → to find step-by-step guides for working with issues.

Removed mentions of color module from IS since it was removed in 10.0.x. Also removed comment about waiting for Yarn 4 to be released since that is also old news.

RE #52

About yarn files which are stored in repo.

1. It allows us not to define which version of yarn should be installed. Should be only node image. Even with pre installed different version of yarn.

In current MR:

docker run -it --rm --name yarn -v "$PWD":/usr/src/app -w /usr/src/app node:20-alpine yarn && yarn build

Will work fine.

And we will don't care about this https://github.com/nodejs/docker-node/blob/62c2e3cfb17ba8d9167b0daebbff9...

2. About `Many sites won't need to run it at all.`
We can say same about https://git.drupalcode.org/project/drupal/-/tree/11.x/core/scripts/css?r...
or a lot of directories that are in core.

yarn.cjs is 2.6Mb. Those other scripts that you have spotted are about 1kb each.

#54 looks like it might have a solution where we can install Yarn purely inside GitLab CI?

I guess it will help ;)

CI mostly fine now

1. Spell-checking command should be configured.
2. .gitlab-ci/pipeline.yml Nightwatch command needs some love.

Regarding committing the yarn-x.x.x.cjs: It's what modern Yarn recommends to do so that you always have a working copy of Yarn. Yes, it's a couple of megabytes, but if it's only in the Git repository to be used for compiling assets and other dev tasks, it can be left out of releases.

The problem is that the release process will have to be modified to take that into account, at present this would be shipped with all future releases.

Asking a bigger question: what does yarn do for us that npm (or perhaps pnpm) does not? They do not depend on living in the repository, as far as I know.

I think this question has own discussion.

Finally brute forced :)

Adding this to review not in terms of code.
(Some DevOps gurus can make it better I'm sure)

But in terms of approach, some js files changes included to MR because yarn linters not silent now ;)

Please review.

nod_ → closed merge request !1932

Why have the JS files been reformatted? IIRC we use prettier to format, so it should be the same if the locked version of prettier is unchanged? You say "yarn linters not silent now", was there a bug somewhere before?

@longwave
Yes for some reason linter failed.
https://git.drupalcode.org/issue/drupal-3109556/-/jobs/334117

I only wanted to pass all tests here. Js linting can be moved to own ticket.

all right, wanted to make sure we could use yarn with corepack only, it's simpler than #54 because we have a package.json with a packageManager entry.

Using yarn like this present some issues:

• Seems like what gets installed by corepack is not validated (or maybe setting the yarn version makes it ok, haven't look deep enough yet)

  Allow checking hashes for exact versions, and check them for default versions #37 and feat: download the latest version instead of a pinned one #134

• When using recommanded yarn init commands, yarnPath is still set to make sure things don't break for people. Meaning that for our users running corepack enable is necessary for the whole thing to work (as shown by the CI scripts needing it)

Not sure if we should go with this or with the executable committed to the repo, in any case it works as expected.

The Needs Review Queue Bot → tested this issue. It fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

This does not mean that the patch needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide → to find step-by-step guides for working with issues.

Is it feasible to do corepack enable in the initial Yarn step and then pass whatever it does as an artifact to the child jobs?

I don't think we can pass whatever corepack enable does as an artifact. Probably a before script makes more sense, I don't have strong preference, whatever works is good with me. I'm more concern of existing devs not having this working. I think we need the yarnPath if an existing yarn exec already exists. Can someone test it?

@longwave:

Asking a bigger question: what does yarn do for us that npm (or perhaps pnpm) does not? They do not depend on living in the repository, as far as I know.

You're right that npm and pnpm don't rely on placing themselves in the repository like Yarn does. It took me a bit of time to adjust when moving from npm. The Yarn documentation does a pretty good job of detailing what it offers. One big difference is that, by default, it doesn't create a node_modules directory at all, but abstracts calls to packages, which are actually stored in archive files; again, the Yarn Plug'n'Play documentation has a lot of info about their architecture and the benefits of it, and of particular interest are performance and predictability, and of particular interest to CI:

Shared installs across disks

Related to the previous point, Yarn PnP allows to reuse the same package artifacts across all projects on the disk. Unlike pnpm, which uses a content-addressable store where each file from each package needs to be hardlinked into its final destination, the PnP loader directly references packages via their cache path, removing a lot of complexity.

what's a recommended way for testing this one?

Believe issue summary has been updated so removing that tag.

@smustgrave

Just need to be runned /core yarn scripts in "node": ">= 18.12" for first MR.

For second MR also should be https://nodejs.org/api/corepack.html enabled

So I applied MR 4698 and just clicked around. Performed some basic tasks like create a content type, view block, etc. Didn't notice anything that broke. Think this may have missed the 10.2 window but maybe get in early for 10.3?

I have only skimmed the Issue Summary. I see that there are two MR on this issue and the remaining tasks states that there is a failing test. However, the tests for the two MR are passing. The issue summary is out of date.

I am tagging this for an issue summary update. In that, include which MR is considered RTBC and then rebase it so it is up to date.

Thanks!

finnsky → changed the visibility of the branch 11.x to hidden.

Rebased both MRs

Seems were some random pipeline failures. Now `3109556-yarn-4-to-11x` branch is ready for review

Yet another approach
https://www.drupal.org/project/drupal/issues/3420041 ✨ Explore opportunities of Bun https://bun.sh/ Active

Seems to have a merge block.

But with 11.x now open wonder if it would be good to try and get in.

finnsky → changed the visibility of the branch 3109556-yarn-4-to-11x-corepack to hidden.

It was easier to create new MR instead of rebase corepack branch.
I think we ready to go with this approach even experimental.

Keeping executable branch for example of approach for review.

I've added yet another MR for tests that js/css/yaml linters will triggered

Mind updating the issue summary with the option you’re going with. And maybe quick sentence for which MR should be reviewed. Or if any can be hidden.

finnsky → changed the visibility of the branch test-css-js-yaml-changes to hidden.

finnsky → changed the visibility of the branch 3109556-yarn-4-to-11x to hidden.

Thanks! Really do think it would be good to get in to 11.x

Some scaffold error here. Not sure how to fix

Seems it is not mine
https://www.drupal.org/project/drupal/issues/3424644#comment-15490289 📌 Update CKEditor 5 to 41.2.0 Fixed

I think we need a change record that tells core developers what to do. I also wonder about the impact of working on 11.x and 10.x - is this going to make it harder?

@alexpott

git co 3109556-4-1-1-corepack && yarn -v
Already on '3109556-4-1-1-corepack'
Your branch is up to date with 'drupal-3109556/3109556-4-1-1-corepack'.
4.1.1

then

git co 11.x && yarn -v                  
Switched to branch '11.x'
Your branch is up to date with 'origin/11.x'.
1.22.21

Switch happends easy.

Was expecting to tag this for docs updates but we actually already tell developers to run corepack enable: https://www.drupal.org/about/core/policies/core-change-policies/frontend... →

A change record is still worthwhile for those that already had Yarn 1 and haven't used corepack yet.

I've just swapped over to using corepack locally and now my core/package.json file changes whenever I run a command. That's really annoying. If we're telling people to use corepack great but we need to update package.json in 10.3 and 10.2 to have the "packageManager": "yarn@1.22" key.

Opened 🐛 Add packageManager key to core/package.json now we recommend corepack enable Active

can't commit this one but RTBC +1

Change record for review: https://www.drupal.org/node/3428571 →

CR reads well, tested on my macbook that it worked

Alright, time to land this. Tested locally back and forth between Yarn 1 and 4 and haven't run into any issues, and we backported the packageManager setting already to previous branches. The changes to .gitlab-ci.yml might make future CI changes trickier to backport, I think we should consider copying those to 10.3.x as well so we have corepack on both sides, but that can be done in a followup.

Committed ad10de3 and pushed to 11.x. Thanks!

Also published the change record.

longwave → closed merge request !6968

Automatically closed - issue fixed for 2 weeks with no activity.

Adding a release note snippet since the CR has important info for core contributors.