[omnibus] Speed up gitlab ci runs

Combining 📌 Get rid of InstallUninstallTest RTBC and 📌 ComponentsIsolatedBuildTest is slow RTBC to see what the effect is.

https://git.drupalcode.org/project/drupal/-/jobs/82114 - 7m49s down from over 10 minutes.

https://git.drupalcode.org/project/drupal/-/jobs/82113 - 8m21s down from over 10 minutes.

Possible next candidates:

Drupal\Tests\config_translation\Functional\ConfigTranslation takes a long time to come back by the looks of things.

I've made some modifications to the pipeline config again:

1. Removed parallel from kernel tests, because often the three kernel test runners end up waiting for gitlab pods which doesn't help them finish quicker.

2. Reduced functional test concurrency down to six - without InstallUninstallTest functional tests can finish quicker/about the same time as nightwatch/functional javascript/build tests.

By doing #1 and #2, the overall pipeline looks like the following:

1. composer and yarn
2. Linting, unit tests, kernel tests
3. All other test types.

We use exactly 10 jobs for #3, which means it maxes out the concurrency limit but also minimises queuing time. However it's only an improvement on the current gitlab MR once those two tests are dealt with, so marking PP-3.

As noted in 📌 GitLab CI integration for core Needs work the Ruby gem parallel_tests has a useful technique we could borrow: it logs test runtimes and then uses that data in future runs to group them into roughly equal runtimes.

Last commits make phpunit parallel 2, and reduce the functional tests parallel by one.

This doesn't improve the overall run time, 14m30s is a touch slower but probably margin of error, however if we're able to resolve the kubernetes pod not available issues, it might speed things up by compressing the blocking unit tests step so that it definitely finishes alongside the lint jobs.

The issue with the parallel_tests approach for us with the current setup is that we're only parallelizing functional tests (and in this issue, unit tests, but those are pretty well matched for times already). The other test types take about as long as the parallelized functional tests and we can't go over ten job concurrency or we end up with queueing.

If we changed the approach and did something like parallelizing all functional, functional javascript, and build tests together, (possibly include kernel tests too) then it might be a further improvement though.

13 minutes.

https://git.drupalcode.org/project/drupal/-/pipelines/20725

With install/uninstall test and the nightwatch composer test optimisations applied, the bottleneck moves to functional javascript tests.

With parallel 2 these take 9 and 7 minutes respectively https://git.drupalcode.org/project/drupal/-/jobs/93117 https://git.drupalcode.org/project/drupal/-/jobs/93118. This is not really an improvement on parallel 1 so it is probably another test that individually takes a very long time.

However nightwatch doesn't use parallel or run-tests.sh and takes about 8 minutes too, we'd need to run those in parallel to get under this, but that seems possible: https://nightwatchjs.org/guide/running-tests/parallel-running.html

📌 Update Nightwatch to 3.x Fixed might help too, the release notes suggest it may speed up tests by up to 25%

Making CI images smaller makes jobs to start faster, so I started to optimize images #3387737: Minimize PHP image size →

Is it worth having another image without apache? For composer, yarn, unit and kernel tests. Or does the overhead of separate images make it not worthwhile?

I have the same question as #17.

There's also an explicit 'start apache' step in the pipeline which we can remove at least for unit/kernel tests which is just a few seconds at most but worth cleaning up, only worth doing if we keep the same images though.

Mysql images has no updates last 2 years so I filed #3388135: Fix base image for Mysql 8.0/5.7 → where few layers are cut off

Optimization of image (first layer is php, apache based on php) drupalci/php-8.2-cli allowed to cut down >60% from 500MB to 188MB comparing to drupalci/php-8.2-apache

btw I can't find any metrics about pod's start-up, so how to measure is open question

Looking for reviews and ideas in #3387737-11: Minimize PHP image size → and related MR

📌 Run nightwatch tests in parallel Needs review should remove the nightwatch bottleneck.

Also incorporating 📌 Distribute @group #slow tests between test runners and mark more tests RTBC .

10m27s with the current patch set https://git.drupalcode.org/project/drupal/-/pipelines/21810

Current state of the combined MR has us at around 11 minutes: https://git.drupalcode.org/project/drupal/-/pipelines/22188

Gonna explore JIT+preload

#3388646: Add option to enable JIT for CI PHP images →
#3108687-36: Compatibility with/take advantage of code preloading in PHP 7.4 →
#2946472-31: Attempt to autostart chromedriver for selenium tests →

Is it possible to skip the ESLint step if no JS files have changed, or the stylelint step if no CSS files have changed?

MR can contain more then one commit so checking for files in all commits is tricky

JIT does not affect PHPUnit but there's some failures in kernel tests https://git.drupalcode.org/issue/drupal-3386474/-/jobs/104981

Biggest thing I'm learning here is that once some of the issues already opened are applied, there is a hard bottleneck of specific long-running tests which break any attempts to optimize concurrency or use parallel runners, and skew perceived performance when CPUs are constrained or concurrency reduced, because they always take a long time to run.

Opened 📌 [meta] Refactor ultra-slow tests Active .

Nothing individually committable from here any more, just merging all the current work into it to see overall progress.

Not sure how hard this would be to split up, but I'm wondering if phpstan (and phpcs) can start while the yarn job is still running. Yarn consistently takes 20-30 seconds longer than composer and phpstan is the longest job in the lint stage by well over a minute.

I am pretty sure that was working until we moved linting to the parent jobs.

Same issue with unit, kernel, and functional tests in the child jobs, they could also start as soon as composer finishes.

They only depend on composer, not yarn, so not clear to me what changed.

11 minutes, 1 second:

https://git.drupalcode.org/project/drupal/-/pipelines/24323

@mstrelan, good point, it used to work like that, issue here: 📌 Use 'needs' instead of 'dependencies' to speed up gitlab CI jobs and run phpstan first RTBC

andypost → closed merge request !7745

2 unit and a few functional tests fail if enable PHP JIT in Apache #3388646-3: Add option to enable JIT for CI PHP images →

andypost → reopened merge request !7745

Added 📌 Convert WebAssertTest to a Unit test Needs review as a child issue

Opened 🌱 [meta] Core test run performance Active to have more of a proper meta for test suite performance improvements, this was more of a sandbox issue so is very messy.

catch → changed the visibility of the branch 3388135-mysql8 to hidden.

catch → changed the visibility of the branch 3386474-11 to hidden.

catch → changed the visibility of the branch 3386474-headers-dev-images to hidden.

catch → changed the visibility of the branch 3386474-jit to hidden.

catch → changed the visibility of the branch 3386474-omnibus2 to hidden.

catch → changed the visibility of the branch 3386474-new-approaches to hidden.

catch → changed the visibility of the branch 3386474-new-approached-printstats to hidden.

catch → changed the visibility of the branch 3386474-meta-speed-up to hidden.

catch → changed the visibility of the branch with-times-multi to hidden.

catch → changed the visibility of the branch 2024-07-testing to hidden.

catch → changed the visibility of the branch 3386474-new-pipeline-world to hidden.