Contrib.social

[meta] Refactor ultra-slow tests

Open on Drupal.org →
Created on 23 September 2023, about 1 year ago
Updated 29 September 2023, about 1 year ago

Problem/Motivation

With 📌 Distribute @group #slow tests between test runners and mark more tests RTBC it's possible to identify the very slowest running test classes (whether unit, kernel, functional, functional javascript) and ensure they're distributed equally between gitlab parallel test runners, and also that they're started first at the beginning of each job.

This ensures that long-running tests finish as quickly as possible. However, there are some tests that take so long to run, that despite starting first, they also finish last.

This undermines optimisations like test concurrency and gitlab parallel runners, because if you have one test that takes 10 minutes, it's going to take 10 minutes no matter how quickly every other tests finishes.

Additionally these tests will significantly impact gitlab resource usage and performance, if we take an extreme but not unrealistic example - if all other tests can be finished in 4 minutes, but one test takes 9 minutes, the runner could have been freed and used for an entire other test run in the meantime.

Optmiizations like converting functional tests to kernel tests (unless it's one of these) are also undermined, because there is a hard floor below which functional test runs can never go below until these tests are dealt with. However, once we've dealt with them, we'll immediately see more benefit from other optimizations too.

Some tests require significant refactoring to improve, like InstallUninstallTest 📌 Add GenericModuleTestBase and use it to test general module things Downport .

Others just have lots of methods, and could be split into multiple test classes with a base class. We don't necessarily need to split tests into method-per-class, it might be that there are logical groupings of 2-3 methods per class, which might be enough to bring them into line with most other tests, can always split them more later if they're still a problem or try to find other ways to optimize.

Steps to reproduce

See 📌 Distribute @group #slow tests between test runners and mark more tests RTBC for how to identify specific slow tests.

Proposed resolution

Biggest outliers identified so far

📌 Add GenericModuleTestBase and use it to test general module things Downport
📌 Split up update semver tests Fixed
📌 CKEditor 5's Functional JS tests take a long time to run Fixed
📌 Split up MediaEmbedConfigurationUiTest Fixed

Possible other candidates:

This is copied from the gitlab output, the number of passes equates to the number of test methods executed, so generally tests with a lot of methods could just be split up.

Drupal\Tests\dblog\Functional\DbLogViewsTest 13 passes
Drupal\Tests\system\Functional\Module\DependencyTest 11 passes
Drupal\Tests\layout_builder\Functional\LayoutBuilderTest 33 passes
Drupal\Tests\demo_umami\Functional\DemoUmamiProfileTest 6 passes
Drupal\Tests\jsonapi\Functional\CommentTest 11 passes
Drupal\Tests\field_ui\Functional\ManageFieldsFunctionalTest 18 passes
Drupal\Tests\taxonomy\Functional\TermTest 10 passes
Drupal\Tests\rest\Functional\Views\StyleSerializerTest 14 passes

Remaining tasks

User interface changes

API changes

Data model changes

Release notes snippet

📌 Task
Status

Fixed

Version

11.0 🔥

Component
PHPUnit  →

Last updated about 1 hour ago

Created by

🇬🇧United Kingdom catch

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

contrib.social Blog FAQ Discussions
Production build 0.71.5 2024