Identify and action solutions for flaky tests

Created on 18 June 2024, 5 months ago
Updated 20 June 2024, 5 months ago

Problem/Motivation

It's very common for me to have to re-run tests that timeout in the gitlab UI. I understand this is largely attributed to slow FunctionalJavascript tests, however I don't recall having core FunctionalJavascript tests being as consistently flaky. (If I'm wrong please call me out.)

If this project is a candidate for core then we should start isolating the flaky tests and working out how to improve them, as there could be some curly problems to resolve.

Example flaky tests (feel free to add).

  • InstallerControllerTest::testInstallUnlockMessage

Steps to reproduce

At this point it's anecdotal evidence (which I invite others to confirm or deny in their own experience) that when you are preparing MRs you often have to rerun specific tests that fail or timeout.

Proposed resolution

πŸ“Œ Task
Status

Active

Version

2.0

Component

Code

Created by

πŸ‡¦πŸ‡ΊAustralia sime Melbourne

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @sime
  • πŸ‡ΊπŸ‡ΈUnited States phenaproxima Massachusetts

    Anecdotally, I find the functional JS tests are flakier than any other kind, but in Project Browser I am occasionally seeing an InstallerControllerTest case fail randomly too.

    But I don't see any random failures with unit or kernel tests.

  • Merge request !520This MR is for testing speed of things. β†’ (Open) created by sime
  • πŸ‡¦πŸ‡ΊAustralia sime Melbourne

    I analysed the events that PM provides using xhprof, and the StatusCheck event is by far the only event where significant amount of time is spent in our tests, which specifally is any tests where:

    • Package Manager is installed
    • drupalGet('admin/modules/browse')

    .

    This is the same as πŸ“Œ Too slow to validatePackageManager() on every page Needs review but obviously the solutions will differ. I've created an MR to try @phenaproxima's suggestion in Slack which is to put an event first and then stop propagation.

    For the record these are the events subscriber that are running when the test his /admin/modules/browse where the vast majority of time is spent:

    • Drupal\package_manager\Validator\ComposerValidator::validate
    • Drupal\package_manager\Validator\DiskSpaceValidator::validate
    • Drupal\package_manager\Validator\WritableFileSystemValidator::validate
    • Drupal\package_manager\Validator\MultisiteValidator::validate
    • Drupal\package_manager\Validator\SymlinkValidator::validate
    • Drupal\package_manager\Validator\StageNotInActiveValidator::validate
    • Drupal\package_manager\Validator\BaseRequirementsFulfilledValidator::validate
    • Drupal\package_manager\Validator\PendingUpdatesValidator::validate
    • Drupal\package_manager\Validator\LockFileValidator::validate
    • Drupal\package_manager\Validator\AllowedScaffoldPackagesValidator::validate
    • Drupal\package_manager\Validator\StagedDBUpdateValidator::checkForStagedDatabaseUpdates
    • Drupal\package_manager\PathExcluder\UnknownPathExcluder::logExcludedPaths
    • Drupal\package_manager\Validator\SettingsValidator::validate
    • Drupal\package_manager\Validator\RsyncValidator::validate
    • Drupal\package_manager\Validator\ComposerPluginsValidator::validate
    • Drupal\package_manager\Validator\ComposerPatchesValidator::validate
    • Drupal\package_manager\Validator\PhpExtensionsValidator::validateXdebug
    • Drupal\package_manager\Validator\PhpExtensionsValidator::validateOpenSsl
  • Pipeline finished with Failed
    5 months ago
    Total: 503s
    #202497
  • Pipeline finished with Failed
    5 months ago
    Total: 2132s
    #202498
  • Pipeline finished with Failed
    5 months ago
    Total: 382s
    #202517
  • πŸ‡¦πŸ‡ΊAustralia sime Melbourne

    Re #2, InstallerControllerTest often calls routes like `admin/modules/project_browser/install-begin/drupalorg_mockapi/awesome_module` and this triggers status checks which can time out. I believe.

  • πŸ‡¦πŸ‡ΊAustralia sime Melbourne
Production build 0.71.5 2024