Account for test type, number of methods, and data providers in test ordering

Created on 25 August 2024, 3 months ago
Updated 21 September 2024, 2 months ago

Problem/Motivation

After 📌 Order tests by number of public methods to optimize gitlab job times Fixed , test classes within a test job will be ordered by number of test methods descending.

Data providers mean that the number of test method runs diverges from the number of test methods, so we currently have to use @group #slow to identify most tests with data providers (at least ones with significant setup tasks) so that they run first.

Steps to reproduce

Proposed resolution

If we did some additional reflection work, we could detect the @dataProvider annotation, and add a 'bonus' to the method count, like +5 or +10 or something. Doing this would allow @group #slow to be removed from those tests which only have @group #slow due to a data provider, leaving just the ones which really are genuinely slow to run (Umami installs et al).

Additionally, if we order by type of test, this should benefit contrib test runs that run multiple types of test in the same job with concurrency (which is how the default template works) - i.e. functionl js tests first and unit tests last.

Remaining tasks

User interface changes

Introduced terminology

API changes

Data model changes

Release notes snippet

📌 Task
Status

Fixed

Version

10.4 ✨

Component
PHPUnit  →

Last updated 39 minutes ago

Created by

🇬🇧United Kingdom catch

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

Production build 0.71.5 2024