[PP-1] Reduce test pipeline times to around 18 minutes and always report back unit tests

This is already done with --with-unit-tests in the core template, closing as by design. Doesn't need its own group

Actually, re-opening with a slightly different purpose.

If we skip the linting tests for unit tests, but add it to all the other types of tests, the following happens:

1. Unit tests run parallel to linting, this removes a 2 minute blocking step in between linting and all other test types so theoretically might reduce overal pipeline time from ~32 minutes down to ~30 minutes.

2. Unit tests will run despite failures in linting tests - this means if you've made a typo in a code comment and also broken a unit test, you'll get the unit test feedback back alongside the linting errors.

All the other test types continue to rely on both unit tests and linting so we don't do expensive test runs for things that are nearly guaranteed to fail.

The MR is based off the main gitlab MR, this is the actual change for this issue: https://git.drupalcode.org/issue/drupal-3386091/-/commit/8cfe4fc5fed6bc6...

Should TESTSDIRECTORY be added to dictionary?

Or phpcs ignore for that file?

bbrala → closed merge request !4731

Tried allowing kernel tests to run parallel to unit tests, it didn't improve things so reverted it.

Small tidy-up merging --with-composer into --with-composer-cache because they were identical.

Last pipeline run was 18 minutes as 5 seconds: https://git.drupalcode.org/project/drupal/-/pipelines/18810

Anything else we've tried is not getting us below 18 minutes, so I think this issue has got as far as it can.

The next stage would be 🐛 \Drupal\Tests\Core\Test\PhpUnitCliTest requires a database and webserver - move to a functional test Active to make the unit test run shorter - that in turn would start all the other test types faster.

Retitling again and updating the issue summary.

We keep seeing the same nightwatch failure over and over on the pipelines, that is the only test type we're not messing around with, so probably not introduced here, but needs its own issue at least.

Also I think we're allowed to bump the priority. gitlabci already completes nearly twice as quickly as DrupalCI, this makes it three times as quick as DrupalCI.

Kernel parallel x 2 gets us down to 17 minutes 24 seconds. https://git.drupalcode.org/project/drupal/-/pipelines/18837 this is because the kernel test runner frees up to pick up the last of the functional tests.

Which gives me the idea to up the parallezation of kernel tests and run them alongside linting again - idea is to max out the 10 concurrent jobs during both the linting and test stages. About to push that.

bbrala → closed merge request !4736

OK 17m49s now: https://git.drupalcode.org/project/drupal/-/pipelines/18855 margin with error with the previous attempt which was 17m30s.

The trade-off is that kernel x 4 can result in one or two of the linting steps queueing, delaying the start of the longest javascript/functional jobs by a minute, so we really are at the limit at 17m30s.

If we can somehow optimize the yarn build step, that would start a lot of things earlier, even 30seconds could shave off 1-2 minutes by freeing up runners. Maybe the cloud caching stuff will help.

The advantage of this approach though is that kernel test failures will report at the same time as phpunit failures and cs failures, all within about 4-5 minutes of pushing to a branch. We still block the properly expensive test runs on unit tests and linting though.

Also now the kernel tests are reporting back quicker, we could consider adding *with-kernel-tests to the more expensive tests without a speed penalty.

OK even at ten concurrency for functional tests, the bottleneck is still the first group of functional tests - there must be some tests in there that are so slow they create a hard floor even while other tests are spread out. So we definitely have reached the limit of overall run times here, but the kernel tests happening earlier isn't affecting the speed of the whole run, so I think that's a good change.

Stopping for now - IMO this is mergeable either as a follow-up to the initial gitlab issue or we could merge it back in there and commit them together.

bbrala → closed merge request !4737

Given that 📌 GitLab CI integration for core Needs work was (and still is) a first proposal and that this issue is built on top of that one, I think the changes from this MR should go directly there and continue the conversation and improvements there.

The MR for 📌 GitLab CI integration for core Needs work already has some feedback and it'd be great to not have two diverging proposals.

The changes to leverage parallelism in this issue are great and the fact that we are already below the 20 minute mark is amazing.

I'm going to mark it RTBC but really aiming for the changes to go directly to 📌 GitLab CI integration for core Needs work and then we can mark this issue as fixed.

Merged over there, marking this as duplicate.