phpstan (next major) failing on setup

Created on 20 September 2024, about 1 month ago

Problem/Motivation

phpstan (next major) fails to set itself up correctly. We get an error in our project despite it never being able to identify any phpstan errors.

1. It should be able to set itself up correctly.
2. Don't flag the tests as failing if the tests were not even able to run. It's a false positive otherwise.

Steps to reproduce

1. Create a merge request for a module project in gitlab. Wait for the Drupal GitLab CI tasks to start running.
2. See the following error on the phpstan (next major) job, paying special attention to the fact that the setup cannot cd to the module code:

Downloading artifacts from coordinator... ok host=drupal-gitlab-job-artifacts.s3.us-west-2.amazonaws.com id=2803537 responseStatus=200 OK token=glcbt-64
Executing "step_script" stage of the job script 00:00
$ SAVED_PWD=$PWD;
$ cd $CI_PROJECT_DIR/$_WEB_ROOT/modules/custom/$CI_PROJECT_NAME && pwd
/scripts-108046-2803541/step_script: line 310: cd: /builds/project/orange_dam/web/modules/custom/orange_dam: No such file or directory


Full job output
Running with gitlab-runner 17.3.1 (66269445)
on gitlab-runner-9dccd6855-vtfgl s8ex1X2yJ, system ID: r_Fs6duF1rd5sK
Resolving secrets
Preparing the "kubernetes" executor 00:00
"CPURequest" overwritten with "2"
Using Kubernetes namespace: gitlab-runner
Using Kubernetes executor with image drupalci/php-8.3-apache:production ...
Using attach strategy to execute scripts...
Preparing environment 00:31
Using FF_USE_POD_ACTIVE_DEADLINE_SECONDS, the Pod activeDeadlineSeconds will be set to the job timeout: 1h0m0s...
Waiting for pod gitlab-runner/runner-s8ex1x2yj-project-108046-concurrent-1-7o3i3233 to be running, status is Pending
Running on runner-s8ex1x2yj-project-108046-concurrent-1-7o3i3233 via gitlab-runner-9dccd6855-vtfgl...
Getting source from Git repository 00:01
Fetching changes with git depth set to 20...
Initialized empty Git repository in /builds/project/orange_dam/.git/
Created fresh repository.
Checking out 5ce54d30 as detached HEAD (ref is refs/merge-requests/63/merge)...
Skipping Git submodules setup
Downloading artifacts 00:01
Downloading artifacts for composer (next major) (2803537)...
Downloading artifacts from coordinator... ok host=drupal-gitlab-job-artifacts.s3.us-west-2.amazonaws.com id=2803537 responseStatus=200 OK token=glcbt-64
Executing "step_script" stage of the job script 00:00
$ SAVED_PWD=$PWD;
$ cd $CI_PROJECT_DIR/$_WEB_ROOT/modules/custom/$CI_PROJECT_NAME && pwd
/scripts-108046-2803541/step_script: line 310: cd: /builds/project/orange_dam/web/modules/custom/orange_dam: No such file or directory
$ INFO_FILES=$(find -L . -name "*.info.yml") || true
$ cd $CI_PROJECT_DIR
$ # If the constraint does not contain ^11 then replace it entirely with ^11 # collapsed multi-line command

*****************************************************************************************************************************
These files have been changed to allow testing for Next Major compatibility:
$ echo "$INFO_FILES" | xargs -I % git diff % || true
$ cd $SAVED_PWD && pwd
/builds/project/orange_dam
$ [[ $_CURL_TEMPLATES_REF == "" ]] && export _CURL_TEMPLATES_REF=$_GITLAB_TEMPLATES_REF # collapsed multi-line command
$ if [ -f $_WEB_ROOT/modules/custom/$CI_PROJECT_NAME/phpstan.neon ]; then # collapsed multi-line command
Executing curl -L --output web/modules/custom/orange_dam/phpstan.neon https://git.drupalcode.org/project/gitlab_templates/-/raw/default-ref/as...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Failed to open the file web/modules/custom/orange_dam/phpstan.neon:
Warning: No such file or directory
100 354 100 354 0 0 3361 0 --:--:-- --:--:-- --:--:-- 3371
curl: (23) Failure writing output to destination
Uploading artifacts for failed job 00:01
Uploading artifacts...
WARNING: junit.xml: no matching files. Ensure that the artifact path is relative to the working directory (/builds/project/orange_dam)
WARNING: phpstan-baseline.neon: no matching files. Ensure that the artifact path is relative to the working directory (/builds/project/orange_dam)
WARNING: phpstan-quality-report.json: no matching files. Ensure that the artifact path is relative to the working directory (/builds/project/orange_dam)
ERROR: No files to upload
Uploading artifacts...
WARNING: junit.xml: no matching files. Ensure that the artifact path is relative to the working directory (/builds/project/orange_dam)
ERROR: No files to upload
Uploading artifacts...
WARNING: phpstan-quality-report.json: no matching files. Ensure that the artifact path is relative to the working directory (/builds/project/orange_dam)
ERROR: No files to upload
Cleaning up project directory and file based variables 00:00
ERROR: Job failed: command terminated with exit code 1

Proposed resolution

1.Fix the phpstan (next major) CI template so it does not fail to set itself up.
2. Modify the exit code so that the merge request is not unfairly marked as failing tests simply because the test setup is buggy.

πŸ› Bug report
Status

Postponed: needs info

Component

gitlab-ci

Created by

πŸ‡ΊπŸ‡ΈUnited States apotek

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @apotek
  • Status changed to Postponed: needs info about 1 month ago
  • πŸ‡ͺπŸ‡ΈSpain fjgarlin

    It's easier if you put a link to the job and/or the pipeline. This job depends on the "composer (next major)" being successful, and we are temporary allowing that job to fail, so it might be that the actual error is in there.

    Please review that first and if you still think it's a phpstan job issue, add the link to the pipeline / MR/ job.

  • πŸ‡ΊπŸ‡ΈUnited States apotek

    Hi fjgarlin,

    Thank you for the response and clarification. It was helpful to note: "This job depends on the "composer (next major)" being successful".

    This makes sense, but the thing is that the composer (next major) job is allowed to fail without failing. But this job marks a big red x failure when it fails. If this job depends on the success of the first (composer) job, then there is some dissonance here. It seems if the first job is allowed to fail, then this job should also be allowed to fail.

    It looks like I will have to disable my CI. It was working fine before, but is now marking the merge requests as failing, despite having no discernible issues is the versions the module has demonstrated compatibility for.

    Overall, it seems odd to fail a module for not having Drupal 11 compatibility when it doesn't even _claim_ to be compatible yet. If the module claimed version 11 compatibility and yet failed the composer (next major) job or failed the phpstan (next major), that would make sense. But in this case, these two jobs are introducing requirements that are not claimed by the module,which means these two jobs are _advisory_ and should not be failing merge requests.

    The specific run is here: https://git.drupalcode.org/project/orange_dam/-/jobs/2803541

  • πŸ‡ͺπŸ‡ΈSpain fjgarlin

    Note that this situation is only temporary.

    The allow_failure will be removed soon: https://git.drupalcode.org/project/gitlab_templates/-/blob/main/includes...

    There is a blog post as well about the transition to Drupal 11 as default, explained here: https://www.drupal.org/drupalorg/blog/gitlab-ci-templates-will-make-drup... β†’

  • πŸ‡¬πŸ‡§United Kingdom jonathan1055

    Hi apotek
    I have been working on issue πŸ’¬ Inform jobs about the "composer" job exit status and bail out early Active which would help in your situation, as this would show more clearly where the problem is. I have just updated that issue with instuctions on how to use it in a pipeline.

  • πŸ‡¬πŸ‡§United Kingdom jonathan1055

    It looks like I will have to disable my CI.

    Or you can just opt out of testing against Drupal 11 until you are ready to do so. Use

    variables:
      OPT_IN_TEST_NEXT_MAJOR: 0
    

    as explained in the blog post linked in #4. All the jobs at the "current" version pass, so you will have a green pipeline result again.

  • πŸ‡¬πŸ‡§United Kingdom jonathan1055

    Thinking about this again, maybe @apotek has a good point. If we are allowing the Composer Next Major to fail during this transition period, than maybe we should have added allow_failure: true to the other 'Next Major' variants also. That would have served the same purpose, to encourage maintainers to see how their module works with Drupal 11, but not impact the pipeline result.

  • πŸ‡¬πŸ‡§United Kingdom jonathan1055

    I have pushed the change mentioned in #7 to MR254 which is ready for review and testing.

  • πŸ‡ͺπŸ‡ΈSpain fjgarlin

    I thought that all β€œnext major” jobs had it, so yeah, it was an oversight.

    I’ll try to review and commit a few small MRs in the coming days (probably from the contrib room at DrupalCon).

  • πŸ‡ͺπŸ‡ΈSpain fjgarlin

    Actually, this behavior might be because of this: https://git.drupalcode.org/project/orange_dam/-/blob/2.x/.gitlab-ci.yml?...

    But I’ll review the MR in any case.

  • πŸ‡¬πŸ‡§United Kingdom jonathan1055

    Actually, this behavior might be because of this

    Yes you are absolutely right. The gitlab templates default in .phpstan-base is allow_failure:true, so the orange_dam project is explicitly setting it to false (which affects all variants). The push I have just committed to MR254 should override that, back to 'true' for 'next major'.

    It would be good if that MR could be committed and a new default release 1.5.9 made, as this will avoid more support requests like this. It may be a while before πŸ“Œ Update templates so 11.0 is the default/current branch RTBC is finally merged.

  • πŸ‡ͺπŸ‡ΈSpain fjgarlin

    Yup, happy to commit quick bug fixes before the big shift. I left feedback on the other issue.

  • πŸ‡ͺπŸ‡ΈSpain fjgarlin

    Closing this one as the work in πŸ’¬ Inform jobs about the "composer" job exit status and bail out early Active will solve the issue.

  • πŸ‡ΊπŸ‡ΈUnited States apotek

    Thank you @fjgarlin and @jonathan1055 for digging in to the issue with me. I appreciate it.

    As a module maintainer, I do appreciate the gitlab CI automations, but given the ad hoc and voluntary nature of module maintenance, it would be good if this tooling defaulted to advisory notices (as agreed in this thread) for anything the module has not yet claimed or delivered upon.

    As a maintainer, I do want my phpstan and phpcs to fail for all versions for which I claim compatibility in my existing code. But I do not want them to fail for versions for which I have not begun the update effort and for which I am not claiming compatibility.

    I also note that since Drupal 11 is out, next major would be Drupal 12. Or am I misunderstanding what next major signifies?

    Thank you again for your contributions. I just wanted to offer some feedback in the issue itself as history. Thank you for #3471235 too.

  • πŸ‡ͺπŸ‡ΈSpain fjgarlin

    You're welcome.

    Some of your questions are answered here: https://www.drupal.org/drupalorg/blog/gitlab-ci-templates-will-make-drup... β†’

    Right now:
    - current is 10, and next major is 11.
    - we are testing next major even if modules don't opt in, in preparation for the D11 shift.

    When πŸ“Œ Update templates so 11.0 is the default/current branch RTBC is merged:
    - current will be 11, and there won't be next major until 12 is available
    - we will only test the versions that maintainers choose to test

  • πŸ‡¬πŸ‡§United Kingdom jonathan1055

    But I do not want them to fail for versions for which I have not begun the update effort and for which I am not claiming compatibility.

    Yes absolutely agree, and we had already catered for Nightwatch and PHPUnit next major, but missed it for PHPStan. That is what we fixed on πŸ’¬ Inform jobs about the "composer" job exit status and bail out early Active so thank you for alerting us the oversight.

    Now that fjgarlin has made a new default release, this fix should be automatically used by all projects running the default gitlab templates version. You can see that this MR has been included by using the two links at the end of this documentation page

    If you run a new pipeline now, it should end green overall even though PHPStan Next Major will show amber in the detailed view.

Production build 0.71.5 2024