Validate stage may use outdated artifact, leading to flaky test results

Created on 22 May 2025, about 23 hours ago

Problem/Motivation

In projects that enable optional compatibility testing using:

variables:
  OPT_IN_TEST_PREVIOUS_MAJOR: 1
  OPT_IN_TEST_PREVIOUS_MINOR: 1

multiple parallel jobs are triggered during the "Build" stage (e.g., `composer`, `composer (previous major)`, `composer (previous minor)`).

From observation, it seems the "Validate" stage (e.g., `composer-lint`, `eslint`, `phpstan`) may use the **first available artifact** from the "Build" stage β€” not necessarily the one created by the default `composer` job.

This could result in "flaky" or misleading behavior, such as running validation tools against a build that used a different version of dependencies than expected.


Steps to reproduce

1. Enable previous version testing by adding:

   OPT_IN_TEST_PREVIOUS_MAJOR: 1
   OPT_IN_TEST_PREVIOUS_MINOR: 1
   

2. Trigger a pipeline with changes that would result in different outputs depending on the version used during `composer install`.
3. Observe which build artifact is used during `composer-lint` or other validate jobs.

Proposed resolution

Ensure that the "Validate" stage uses the **artifact from the default `composer` job**, rather than the first available build artifact. This could involve making job dependencies more explicit or restructuring how artifacts are passed.

Remaining tasks

- Confirm current artifact selection logic in GitLab CI config.
- Modify `validate` jobs to depend only on the default `composer` job if appropriate.
- Add comments or documentation to clarify behavior for others.

User interface changes

None.

API changes

None.

Data model changes

None.

πŸ’¬ Support request
Status

Active

Component

gitlab-ci

Created by

πŸ‡ΊπŸ‡ΈUnited States trackleft2 Tucson, AZ πŸ‡ΊπŸ‡Έ

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Production build 0.71.5 2024