Update templates so 11.0 is the default/current branch

Makes sense to me to postpone this for a few weeks. We could pick a half-way point between 11.0 and 11.1 like October 1st maybe?

I agree with this. I kept thinking about it when we first talked about it and I agree that it'd be more disruptive than helpful to do the jump straight away, so waiting a few weeks seems easy enough and sensible at the same time.

There is an issue about LTS here #3444714: Support LTS version of Drupal → . I don't know if it will be needed or if, once we jump, we keep the previous major to the LTS version.

Yes there is really no benefit at all in pushing quickly for D11 to become the default test version. We have a good infrastructure for testing against D11 now, which is opt-in and non-disruptive. There are many of the D11 Automatic Bot issues and MRs which are being actively worked on. Also those issue forks are used for testing Next Major when that module is a test dependency. See 📌 Gitlab PHPunit job at Next Major needs D11-compatible test dependencies Fixed where we have a replacement temporary composer.json for Next Major which adds the repos and forks of the D11 issues. It is an interim solution as things are in flux with other modules, but that is OK because its not the 'current' branch test.

At some point we could ease in to encourage testing at D11 by simply switching the default for OPT_IN_TEST_NEXT_MAJOR to 1 as a temporary measure during the transition. Then we set it back to 0 when D11 becomes current and we have the 'real' next major.

At some point we could ease in to encourage testing at D11 by simply switching the default for OPT_IN_TEST_NEXT_MAJOR to 1 as a temporary measure during the transition. Then we set it back to 0 when D11 becomes current and we have the 'real' next major.

This also seems like a great idea - maybe a couple of weeks after 11.0.0 for that one?

To confirm:
The plan would be to update all the variables when 11.x is released and separately (in this issue) change DRUPAL_CORE for all the Composer jobs correct?

While the suggestion could be less disruptive to the community it also feels like a break in our job API promise to me. Next Major is promised to be a next major, current is promised to be current major and previous major is previous major.

This proposal feels like it is trying to fix a fault in our core release timing (too short of a time sync between API lock in Beta1 and Stable release) or an issue with inactive maintainers by delaying updating testing.

IIRC we did have at least one incident (might have been D7 related) where Core threw errors because it was not latest that did break tests when it was not expected.

If a contrib module did a similar check to make sure that the current major release is actually the current major (maybe the module feared that we would updating and added their own defense) this could force maintainers to make a manual change to their CI files.

I will also note that until #3458238: CI installs wrong Drupal version → lands we do not always respect the DRUPAL_CORE target value (although in this case it may not pose a significant issue, we may just end up testing D10 and D11 with the “current major” job depending upon the project).

The following suggestion is not a fix today unless we open up a 2.x branch of templates however maybe instead our “current major” job should be replaced with a “max dependencies” job where we let composer find the latest compatible versions, including core (be it D9/D10/D11) and just run from there.

Yes, I bet that we will need to update some of the variables in one batch, when new versions are available (the check_versions job might even start failing if we don't) and then use this issue to "promote" or "push" all the testing so that "current" is D11 and "previous major" is D10.

We are getting the failure now as Drupal 11.0.0 is out there: https://git.drupalcode.org/project/gitlab_templates/-/jobs/2311261

We will ignore these for now until we change the values. I'm not sure that we should do something for the "check_versions.php" script in the interim.

I've put together the changes for when we do the switch. It's in https://git.drupalcode.org/project/gitlab_templates/-/merge_requests/242

Most notably current versions will be 11-based, previous versions 10-based. Anything next is still 11.x-based as that is the equivalent of "main".
That also means that we are no longer testing anything D9 unless maintainers change the variables themselves (which is totally possible).

Still to fix:
- PHP versions (minimum 8.2 for 11.0.x) 🟢
- SQLlite version ( #3447792: See if we need to raise sqlite version for Drupal 11 → ) 🔴
- Downstream projects should be green:
-- API: 🟢
-- Decoupled pages: 🔴 (only one nightwatch test)
-- Key CDN: 🟢

Updated title and description slightly.

MR needs review in any case.

We will ignore these for now until we change the values. I'm not sure that we should do something for the "check_versions.php" script in the interim

Per comments #6/#7 I though the plan was to update the values today and use this issue to “roll back the impact” to tests by overriding core target per job.

Updated summary with todo tasks.

We need to have all the dowstream projects sorted before rolling out any changes. What I meant in #7 was that we might need to update some variables if something was really broken, but it's "just" our script to check_versions, nothing else. So that leaves this issue to take care of all the rest.

I've updated the issue description with things to do in this issue before it is merged.

Updated TODO.

Updated completed tasks.

That also means that we are no longer testing anything D9 unless maintainers change the variables themselves (which is totally possible).

This alone is a reason to wait -- most modules should still be supporting Drupal 9.5 (~33% of D8+ sites are still on D9 as of Aug 1st 2024), and dropping test coverage means that seemingly innocuous changes (looking at you, phpcs issues) can break these sites and there would be no warning to the maintainer.

I could write a whole blog post about this -- but generally, IMHO modules (and thus the testing suite) should support three major versions of Drupal. Its better for the modules and testing suites to be -behind- the adoption curve to ensure stability and upgradability for users. Especially since those on older versions are likely larger, more complex sites. As a community we should strive to make the upgrade process easier, which means supporting a wider set of versions. That said, for Drupal 11 readiness, dropping Drupal 8.9 support made sense as very few sites were still on it, and there comes a point where old support should just end.

Brainstorming out loud here: The question then becomes naming and timing. For the next year we'll have core support like this:

Aug '24: 10.2, 10.3, 10.4, 11.0, 11.1
Q4 '24: 10.3, 10.4, 10.5, 11.0, 11.1, 11.2
Q2 '25: 10.4, 10.5, 11.1, 11.2, 11.3
Q4 '25: 10.5, 11.2, 11.3, 11.4, 12.0

Looking at this schedule, I think it makes sense that 9.5 remains in the testing mix until Q4 '25, which would be when D12 is starting work.

However this also introduces a lot of testing scenarios, and thus once Drupal 10.x goes into LTS phase, I don't think we need to test all of these versions. Once 10.4 comes out as the LTS release, we can probably reduce our D10 window from 10.2, 10.3, 10.4 to simply 10.4. THIS is when Drupal 11 should become the current version of Drupal.

Therefore, I propose there are 3 times where a job pipeline change should occur in the test scenarios:

• n.1 release:
  This is kickoff for the lifecycle of the current major version of Drupal. This corresponds with the previous release entering LTS phase.
  --> NEXT_MAJOR_MINOR (11.1) to CURRENT
  --> NEXT_MAJOR (11.0) to PREVIOUS_MINOR,
  --> NEXT_MINOR (10.4) to PREVIOUS_MAJOR,
  --> PREVIOUS_MAJOR (9.5) to PREVIOUS_LTS.
  The NEXT_MAJOR and NEXT_MAJOR_MINOR pipelines get disabled.
• n+1 alpha release:
  This phase is when we're running readiness for the next version. At that time, PREVIOUS_LTS tag gets disabled and NEXT_MAJOR tag gets enabled with the next version of Drupal.
• n+1 .0 release
  The only change needed when a new major release occurs is the addition of NEXT_MAJOR_MINOR. This allows projects to test against 11.1, but still allowed to fail to support contrib module catch-up.

Updated issue description.

--

Re #18. We are waiting, and we are trying to get the best strategy and to get all the dependencies in place first. Once that's done, we will offer information so maintainers can decide what to do.

We have been running during all these months D9 + D10 (current) + D11. The shift, when it happens, would still cover Drupal 10, until Drupal 13 is released. For example, a lot of maintainers were running "current" + "next major", and when the shift happens, "current" + "previous major" would be equivalent.

Drupal 9 is EOL since December 2023. There will be options to continue testing "old" versions if desired, as we are not limiting anything. We are providing recommended defaults. You can pin the version of the templates that you are using to a static version that won't change, you can override any part of the template, you can create variants easily, so I think that there will be options suitable for the different cases.

📌 Automated Drupal 11 compatibility fixes for decoupled_pages RTBC needs to be merged. I RTBCed the issue and pinged the maintainers via slack.

I've tested this MR with Scheduler and it basically runs, but I have some questions. For ease of viewing, the version info can be seen at the end of the composer jobs.

1. For current, max php and next minor we get 11.0.x-dev a17b3d3 but for Next Major we have 11.x-dev c03b98e. Is think this difference is expected, is that correct?
2. If #3426289: Cater for empty next/previous minor/major when it's not possible → were done it would have a beneficial effect, because 'next minor' testing is a waste of resource right now
3. Previous Major is 10.2.7 but Previous Minor is 10.2.2 (older than previous major). I guess this is also a variant which is not applicable immediately after the switch to D11

I have separated out the TODO tasks into technical work and actions. These can be expanded as and when, and a timetable worked out. But for now we are concentrating on the technical work. I will test the deprecated reference next.

Fix list syntax. The "preview" does not work properly and does not show this type of formatting error.

Re #21.

1. Yes, those values seem correct.
2. Agree that doing this as well would be ideal to avoid the cases in 1, where three variants run the same thing.
3. Right now we are using the last (previous major) security release as opposed to the last (previous major) stable release. Happy to change this as we also have the LTS issue where the last (previous major) security release might make more sense.

Thanks for tidying up the issue description and for the additional testing.

As discussed I have added a check for $CORE_PREVIOUS_STABLE into check-versions.php. I also reverted the value back to the wrong one we found, so the pipeline job should fail.

As expected, the job failed https://git.drupalcode.org/issue/gitlab_templates-3463894/-/jobs/2369415 showing that $CORE_PREVIOUS_STABLE 10.2.7 should be 10.3.1

Setting to NR as, in addition to the code changes, we now have a draft-announcement.md" class="!text-blue-400 !no-underline" target="_blank" rel="nofollow" >draft announcement →

I have cerated MR 246 for step 1 of the process, to change the default of OPT_IN_TEST_NEXT_MAJOR to 1 temporarily.

📌 Automated Drupal 11 compatibility fixes for decoupled_pages RTBC deployed.

#3447792: Add _TARGET_PHP_IMAGE_VARIANT to cater for newer sqlite versions for Drupal 11 → (and ideally #3458238: CI installs wrong Drupal version → ) to go.

#3447792: Add _TARGET_PHP_IMAGE_VARIANT to cater for newer sqlite versions for Drupal 11 → merged. This was the last technical blocker.

#3458238: CI installs wrong Drupal version → is still under discussion, but it's not a blocker.

We will review the announcement blog post this week and proceed with https://git.drupalcode.org/project/gitlab_templates/-/merge_requests/246

Updated the issue description with the current situation.

I commented on https://git.drupalcode.org/project/gitlab_templates/-/merge_requests/242... but not sure if you see it as the thread is closed. The change to allow 'Composer Next Major' to fail should be in MR 246 and then reverted in MR 242. I can make this change if you confirm that I've understood it correctly.

You are right @jonathan1055, the change was meant to go in the other MR, and I just added it. I don't think it'll need to be reverted, as we were already allowing failure in PHPUnit and nightwatch jobs for the "next major", so this way it'll be more consistent. The error will show, but the overall pipeline result won't fail if any of the future jobs ("next major") fail.

I don't think it'll need to be reverted, as we were already allowing failure in PHPUnit and nightwatch jobs for the "next major", so this way it'll be more consistent.

You are right that it is "more consistent" but it also means that if there is a failiure in the Composer Next Major then the PHPUnit and Nightwatch Next Major jobs will still be attempted (even though they will almost certainly fail because the Composer Next Major failed). I think we left 'Composer Next Major' without being allowed to fail so that resources are not wasted in running the later jobs. Also, if the Composer job is allowed to fail then attention is taken away from it, and devs try to work out why the phpunit job is failing (I did this and wasted some time on it, only to realise that the problem was the Composer job instead).

So it's not clear cut which scenario is better. Happy to go with your preference, but just pointing out the other side of the argument.

Can we test in a pipeline what happens if the "composer (next major)" job fails, whether the "phpunit (next major)" and/or "nightwatch (next major)" run at all? I'd assume that they do not run (thus not wasted resources) but we'll clear any doubt with a test for this case.

I tested with this:

test a:
  stage: test
  allow_failure: true
  script:
    - exit 1

test b:
  stage: test
  allow_failure: true
  needs:
    - "test a"
  script:
    - exit 0

See the result here: https://git.drupalcode.org/project/gitlab_templates/-/pipelines/260356

It does seem to run. So yeah, let's add a note to revert this change.

Nice test :-) That's what happened to me when I started testing at Next Major. The phpunit job ran, failed, and I spent some time trying to debug it, not realising that the fault was in Composer Next Major. Even though it is Amber, your attention is drawn to the red phpunit jobs. If there was a way to tell a job that its needs: pre-requisite has to end with 0 before it can run, that would be good. Then we could allow the Composer job to fail.

I think that's the difference between needs and dependencies but I'm not sure. Maybe something to play with in another issue.

https://www.drupal.org/drupalorg/blog/gitlab-ci-templates-will-make-drup... → is now published.

I added extra logic to allow for empty variables on certain variants as a result of #3426289: Cater for empty next/previous minor/major when it's not possible → , which will be committed soon.

There are four variables that can be blank, you catered for three of them, but missed CORE_PREVIOUS_STABLE, so I added that. I removed the exception that allowed CORE_NEXT_MINOR to be the same as CORE_SUPPORTED. This is not needed now that CORE_NEXT_MINOR can be blank instead. Also did some more improvements to the script and the test values.

Thanks for the additions to the script @jonathan1055.

--

As per https://www.drupal.org/drupalorg/blog/gitlab-ci-templates-will-make-drup... →
- Merged MR246 (Temporarily turn OPT_IN_TEST_NEXT_MAJOR on for all contrib)
- I will release tag 1.5.6 later today.

#3458238: CI installs wrong Drupal version → could be merged after it is RTBC.

#3462681: Add W3C compliant JS testing → should ideally be sorted before we do the final switch.

Great news that we have completed step 1. I've had some more thoughts on the timing and plan for steps 2 and 3, and even though I know expected dates have been published in the blog, I thought I would at least share my ideas in case we want to revise them.

We currently have flexibility in when to commit step 2, but once that is done the clock is ticking on the gap between that and step 3 because step 3 only involves moving the default-ref tag. In practice this means we cannot make any other important bug fix and release it to all Contrib during this time without step 3 also being bundled in with that change. It would therefore be beneficial to minimise the chances of major bugs being found after step 2 is completed, which suggests that we allow more time in the step 1 phase (where we are now) before commiting step 2. We may see an upturn in support requests in the next few days now that all Contrib is testing against Drupal 11. This may involve making some fixes to the templates, or at minimum it may take time and effort away from preparing for step 2.

The current planned date is Thursday 5th September (7 working days from today). Maybe we should put this back a bit more, say another week? This would give more time for Contrib to find any problems in the templates at Next Major, give us time to resolve them and release 1.5.7 say. It would also allow us to finish #3458238: CI installs wrong Drupal version → and #3462681: Add W3C compliant JS testing → before step 2, both of which would be beneficial.

Agree. That's why we used the word "expected".

It'll depend on whether new issues/bugs are reported in the next week or two. But yeah, I think that these two are now requirements before Step 2 happens:
- #3458238: CI installs wrong Drupal version → (nearly there)
- #3462681: Add W3C compliant JS testing → (not there at all, and this one is more crucial)

I'd say let's focus on the above two first, watch the queue, and play it by ear. We have some flexibility in the dates and we don't need to force things, but also, if everything goes well, we can also stick to the plan.

I updated the issue description with the above issues.

Yes that sounds like a good plan.

Regarding the blog post, does anything have to be done/requested for it to be inclued in the weekly e-mailing of the Drupal Newsletter? I don't know if they are sent out all at the same time, but mine last was Thursday 22 August, (one day after the blog post was published) and it did not contain anything about it. The next would be Thurday 29th, and it would be good to have this included.

I made a request to our marketing team to see what they can do about it. We don't control "The Weekly Drop" but we sometimes send them blog posts for consideration. "The Drop Times" already posted about it.

Your request worked, it was included in "The Weekly Drop" mailed today
https://mailchi.mp/drupal/20240829?e=2de42d4404

One of the blockers was committed now. Updating description.

The other blocker is ready for review.

I have made some more corrections to the blocker #3462681: Add W3C compliant JS testing → and it is ready for review again. Hopefully some of the 18 followers of this issue could take a look at that one. I do not have the expertise to verify the W3C testing parameters or new driver, all I have done is make the pipeline jobs work as per the intended changes requested, and checked them with a javascript test which runs (and passes) on both the legacy and new driver.

I've now rebased with the merged issue #3462681: Add W3C compliant JS testing → .
We need to address "@todo" referring to the current MR, so setting back to "Needs work". @jonathan1055 mentioned that he's ok taking over these changes.

I've done the @todo work. When I was making the changes in MR242 I was fairly careful to make @todo notes on all the bits that needed to be undone. So I hope I've got it all.

Tested on Scheduler and the PHPUnit javascript tests now fail with the new driver (in PHPUnit current, which is D11 in this MR)
https://git.drupalcode.org/project/scheduler/-/jobs/2778051

But a few days ago (when testing MR242) the same tests passed at Next Major (which was D11 back then). You can check the driver info as I had CI_DEBUG_SERVICES enabled.
https://git.drupalcode.org/project/scheduler/-/jobs/2738867
I do not know why this should be, and was expecting these to pass. Is there some instabilty when using the Selenium driver? Have things changed between 13 Sept and today?

Anyway, this is ready for review, but not urgent at all, now that we're not doing the big shift until sometime after 27 Sept.

There is another odd thing that I observed but can't explain yet. I added variable
SYMFONY_DEPRECATIONS_HELPER: 'ignoreFile=/builds/project/scheduler/web/core/.deprecation-ignore.txt' to all the PHPunit jobs, and the "PHPUnit Previous Major" (D10, with legacy driver) Scheduler API test failed as expected, becuase there is a deprecation in the code when running at 10.3
https://git.drupalcode.org/project/scheduler/-/jobs/2778595

But the "PHPUnit current" (D11 with new W3C driver) did not fail at all, no deprecation and the test passed.
https://git.drupalcode.org/project/scheduler/-/jobs/2778593

I don't know if this is wrong but it surprised me. Is that deprecation somehow being ignored in D11 when it is exposed in D10.3? Again, not urgent, I am not asking for answers, just reporting what I have seen.

Thanks for addressing the @todos. The code is looking like I expected. I triggered the downstream pipelines here: https://git.drupalcode.org/issue/gitlab_templates-3463894/-/pipelines/28...

As for the issues above. I can't see those jobs even after getting push access to the issue they are linked to. I guess the test a few days ago for this MR used the legacy driver and now uses the new driver.

Core is currently pinning the versions here: https://git.drupalcode.org/project/drupal/-/commit/9734ca5f39d1546198d94..., even thou they are planning to revert it. They did it because they had random unexpected errors, so maybe it's related.

Also, "decouple_pages" nightwatch tests don't seem too happy and they were working on the "next major" for #3462681: Add W3C compliant JS testing → .

Maybe there is some mix-up between legacy vs new driver in some jobs?

I don't know if this is wrong but it surprised me. Is that deprecation somehow being ignored in D11 when it is exposed in D10.3?

All of phpunit's E_USER_* handling completely changed and Drupal 11 had to rewrite our deprecation handling to match, so it is quite possible that it is different, either a bug or something extra needed in phpunit.xml.

Nightwatch tests regularly fail in core HEAD so it may be increased randomness.

Note that core ran into 📌 Fix selenium performance/stampede issues in gitlab config and BrowserTestBase Needs review which required divination (endless trial and error running pipelines) to figure out, so anything weird like that pops up, worth reading that issue first to save some of the same pain I went through.

Thanks for that info @catch about the deprecation processing. I will try to find what happend to that partcular deprecation.

For Decoupled Pages I noticed that it passed at 'next minor' even though it failed at 'current' and these should be broadly similar, as they shold both be using the new W3C Selenium driver. So I extracted and compared the raw logs, and found the following differences, which probably explains it:

Nightwatch (current)

Drupal: 11.0.5-dev
Nightwatch: version: 2.4.2

Nightwatch (next minor)

Drupal: 11.0-dev
Nightwatch: version: 3.7.0

100%. New driver is for nightwatch 3+ as far as I know. I guess that means that we should have the "current" one run the legacy driver for now. I'm not sure if the nightwatch jump will go in 11.0.6 but we can add a note/todo to the code to reflect this.

So is it the case that the NIghtwatch version is built in to the Drupal image we use for each variant? I cannot see any place where we control that separately. If that is the case then do we have to set the legacy driver or W3C driver based on our knowledge of what Nighwatch version is used where? or is there some way to detect, say in the Composer job, what Nightwatch version is being used, and set the driver accordingly. Otherwise, it sounds like we'll have to keep a watch on when it might change. If it can't be automated in "composer" maybe we can add an extra check in 'Check Versions' to alert when it needs to be changed?

I don't think we can automate it because it needs one service or the other and we can't add conditionals in that part of the job.

Hopefully, this will be a few weeks/months where we need to keep an eye on it. Hopefully, soonish it'll be:
- D10 (previous major): legacy unless it's backported
- D11 (current): new driver
- D11+ (next minor): new driver

OK I can amend the MR. It will mean putting back some of the @todo which I removed in #57, but that's not a problem.

Changes made. I raised 📌 Update to new driver when D11 "current" uses Nightwatch 3 Postponed and created MR257 so that we can refer to this in the @todo comments.

Tested on Scheduler with every variant enabled (except of course that 'Next Major' does not exist yet).
All drivers and variables seem to be what we now think we need.
https://git.drupalcode.org/project/scheduler/-/pipelines/287354

Could you trigger the downstream pipelines?

I made another observation that we should not be testing at "previous minor" 10.2.2 when we have "previous major" set to 10.3.5. So I added a check which produces - [ERROR] CORE_SECURITY_PREVIOUS_MINOR (10.2.2) should be blank. in this scenario.

Thanks so much for the latest changes. I’ll try to review them tomorrow.

I have edited #68 to give some more details.

Also I forgot to explain another improvment I made to the 'Check versions' output, to make it easier to see what variables and values we are using.

Old

New

Thanks so much for the improvements!! It looks much better and I agree that previous minor should be empty right now, so good call there.
Downstream pipelines: https://git.drupalcode.org/issue/gitlab_templates-3463894/-/pipelines/28...

The code looks good to me. I'm going to give it a tentative RTBC knowing that we will not deploy this until DrupalCon has passed.

Nice to see Decoupled Pages Nightwatch tests all pass. The new current Drupal: 11.0.5-dev has Nightwatch 2.4.2 with the legacy driver, and 'next minor' with Drupal: 11.0-dev has Nightwatch 3.7.0 and the new Selenium driver.

We’re merging a few issues today. This will need rebasing.

Thanks for rebasing the code from the other issues. I believe that this MR is in a good state now, right? I might just merge 📌 Ignore all git files in artifacts Active before, but other than that, I think this is the main outstanding issue, and the code seems to be good.

We know about the legacy vs new driver situation, which can change if 📌 Use selenium/standalone-chrome instead of our chromedriver image Needs work is backported, but for now, it is not, so again, this MR should be good.

I was going to set it back to RTBC but just wanted to double check before doing it.

There are no more blockers.

I updated the versions for the new releases.

I have updated the dates for steps 2 and 3 in the blog post → .
- Step 2 (merging this MR and creating release 1.6.0) will happen on Monday, October 7th, 2024
- Step 3 (making 1.6.0 the default for all contrib) will happen on Monday, October 14th, 2024

RTBC.

Updated I.S. with dates and 1.5.9 version

🐛 11.0.x yarn dependencies have mushroomed Active - created today and it's a blocker for this.

🐛 11.0.x yarn dependencies have mushroomed Active is now RTBC. I will merge this the moment that that issue is merged.

Just commiitted 🐛 11.0.x yarn dependencies have mushroomed Active .

I merged this now! This is step 2. Only step 3 is left, which will happen next week.

I’m thinking about including 🐛 upgrade_status job fails due to unmet database requirements Active and 📌 Use stable core versions Active , tag 1.6.1 and then make that the default for all contrib on Monday (Step 3).

I can also set 1.6.0 on Monday as planned, and merge those two anyway to "main", but I think they’re good additions before the big switch.

That sounds like a good plan, and we can add 📌 Better failure if there are no variables Active too.

Blocker before step 3 (and final): 📌 Review D7 pipelines with D11 shift changes Active

We think that it can be solved today so it's only pushing the expected date by one day only.

The above blocker is now fixed / merged, so I will do Step 3 tomorrow morning.

Updated summary with latest tag and date for step 3

1.6.3 is now the default tag for all contrib.

Thanks all!

This has lead to tests failing; see 🐛 Stop changing the meaning of tests Active .

Automatically closed - issue fixed for 2 weeks with no activity.