Assert known preconditions for test runs and fail early if unmet

Created on 8 November 2022, over 1 year ago
Updated 14 September 2023, 10 months ago

Problem/Motivation

Our automated tests have implicit dependencies (i.e., not documented or asserted, if even well-known) on apparently arbitrary environment configuration details, leading to tests failing in unpredictable ways depending on where they're run. The result is developers waiting for long test runs that are doomed to fail from the start and then getting cryptic and misleading error messages that don't help them identify or solve the actual problem.

Steps to reproduce

Proposed resolution

We should identify ambient or environment-level preconditions for tests to run and assert their correctness at the beginning of test test runs and exit early with helpful, explicit error messages. (Eventually, we should eliminate as many such dependencies as possible, but let's start here.)

Remaining tasks

  1. β›” Identify as many environment-level test dependencies
  2. β›” Assert their correctness at the beginning of test runs and, if unmet, fail early with clear error messages explaining how to fix the problem
  3. β›” Create a follow-up issue to eliminate as much of the environment-sensitivity as possible
  4. β›” Create a follow-up issue to ensure that there's parity between production requirements and test preconditions. (We don't want tests to require different conditions than production code!)

User interface changes

API changes

Data model changes

Original report

In #3319497-6: Build test failure on local in 8.x-2.x due to drupal core packages being added to vendor.json β†’ and #3319497-7: Build test failure on local in 8.x-2.x due to drupal core packages being added to vendor.json β†’ , we discovered that \Drupal\Tests\package_manager\Build\TemplateProjectTestBase::createVendorRepository() behaves in an environment-dependent way.

Once AU moves into Drupal core, it's unacceptable for its tests to not work on environments with certain additional packages installed in a certain way.

That'd violate the testing gate β†’ .

The patch in #3319497-6: Build test failure on local in 8.x-2.x due to drupal core packages being added to vendor.json β†’ provides momentary relief to unblock the people working on AU, but does not provide a systemic solution yet. This issue exists to do precisely that: solve it systemically. Or, at the very least, fail explicitly.

πŸ“Œ Task
Status

Fixed

Version

3.0

Component

Code

Created by

πŸ‡§πŸ‡ͺBelgium Wim Leers Ghent πŸ‡§πŸ‡ͺπŸ‡ͺπŸ‡Ί

Live updates comments and jobs are added and updated live.
  • Documentation

    Primarily changes documentation, not code. For Drupal core issues, select the Documentation component instead of using this tag. In general, component selection is preferred over tag selection.

Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

Production build 0.69.0 2024