Distribute @group #slow tests between test runners and mark more tests

Left a comment in the MR

But seems the pipeline failed

Started a new MR at https://git.drupalcode.org/project/drupal/-/merge_requests/4837 with a cleaner history and more selection @group #slow addition.

Ready for review now.

This is easiest to see the results of on https://git.drupalcode.org/project/drupal/-/merge_requests/4802/pipelines where some successful pipelines are finishing in ~11 minutes.

The easiest way to tell that this is working is in FunctionalTests jobs like https://git.drupalcode.org/project/drupal/-/jobs/106009

We have enough tests that are sufficiently slow that even when they run first, they still finish last, as long another slow test isn't started at the last minute - all the other tests then fit in the middle.

📌 Add an automated way for slow tests to fail drupalci builds Needs work doesn't help us in its current state because it measures individual method test time, and on gitlabci (and DrupalCI) tests are run in parallel by class. So a test with 15 methods taking 50 seconds each can slip under a 1 minute time limit for individual methods.

The @group #slow tag annotation then gives us a list of targets for test optimization too.

I added a small nit with a space in one of the files other than that the logic is sound to pad the slow group before.

RTBC

Reran pipeline as this was showing a fail

Pipeline looks to still be failing? https://git.drupalcode.org/project/drupal/-/pipelines/22710/test_report

Ooh, rerun one job nice, trying that

Failed again https://git.drupalcode.org/project/drupal/-/pipelines/22710/test_report - do we have an issue in HEAD?

There's an issue but no fix yet 🐛 [random test failure] Random failure in MigrateBlockContentTranslationTest Needs work . I wonder if this issue is exacerbating the random failure.

I've moved the Kernel test changes over to 📌 Mark more KernelTests with @group slow Postponed in the hope that resolves the persistent random test failure here, so that the random test failure doesn't block this one going in -

OK it's not actually marking more kernel tests with @group slow that's the problem but for some reason the way the order changes with distributing them to the start of each job. Since there's only one kernel test job it must be an ordering issue. Trying with that removed. That means we might be fine doing all the tagging here and moving the run-tests.sh change to a follow-up.

I don't get it, reverting everything except the functional tests still leaves the kernel failure. Maybe 🐛 [random test failure] Random failure in MigrateBlockContentTranslationTest Needs work is just much more frequent that it's failing every recent run on this issue.

The last green run against HEAD had 30277 tests https://git.drupalcode.org/project/drupal/-/pipelines/23154/test_report but with this branch there's only 30182 https://git.drupalcode.org/project/drupal/-/pipelines/23167/test_report

The difference is in the number of functional tests

HEAD: 4271
This branch: 4176

So about 50 less functional tests are run.

Do we know why that is different?

I rebased against 11.x in case that's it, hopefully the 9 commits it was behind explain it

Still 30182 after merging head https://git.drupalcode.org/project/drupal/-/pipelines/23206/test_report

Found the issue and committed a fix, $array1 + $array2 vs. array_merge($array1, $array2) - it was dropping the non-slow tests with the first couple of array keys.

MR: https://git.drupalcode.org/project/drupal/-/pipelines/23298/test_report (30278)

vs.

11.x https://git.drupalcode.org/project/drupal/-/pipelines/23269 (30278)

Remarking.

Committed to 11.x and backported to 10.1.x, thanks!

larowlan → closed merge request !4837

Nice!

Next step is 📌 [meta] Refactor ultra-slow tests Active .

Automatically closed - issue fixed for 2 weeks with no activity.