[CI] Report PHPStan baseline statistics in job

Nice! lets do it. https://git.drupalcode.org/issue/drupal-3499245/-/jobs/3994661

No wait, shouldn't we expose this as a metrics report?

https://docs.gitlab.com/ee/ci/testing/metrics_reports.html

This would make this visible in the gitlab interface.

And an example:

https://gitlab.com/gitlab-org/ci-sample-projects/metrics-reports

File format is extremely easy

Did a. test here: https://git.drupalcode.org/project/deprecated_code/-/merge_requests with a mr to main and 1.x with different metrics.

Few things to note:
Metric changes are based on the target branch of the merge request, which means this will work pretty quite well to show the correct metrics based on the main branches (11.x, 11.1.x etc) without the need of any weirdness or logic.

Seems also to be provable with a MR locally, but since the current job only submits artifacts when the job fails it might need a little more massaging. So lets follow up on that after this issue.

I've checked the reporting thingie, it seems reports are always uploaded so my assumption (and therefor the use of a follow up) is invalid. I'm going to update the mr here to include the metrics for easy viewing.

Nice! If we do this, we need however to have the stats calculated right also (and especially!) for the baseline resulting from changes in the MR. We need to get it right first, though, ✨ [CI] Generate full baseline upon PHPStan failure Active .

@bbrala how will the delta be calculated though? i.e. suppose my MR reduces the baseline errors from x to y: how will we get the info that there was a change from x to y?

I think i mentioned that earlier. It uses the stored metrics from the target branch of the mr as a base and shows the diff compared to that.

Ah thanks, but where are they stored in the target branch? Not as a file in the git repo, right?

As an artifact on gitlab :)

Gotcha. Thanks!

✨ [CI] Generate full baseline upon PHPStan failure Active got committed, this is actionable now.

I love i see metrics report now in the mr. Marked as new as expected :)

That's so cool https://git.drupalcode.org/issue/drupal-3499245/-/jobs/4970368

Lets definitely add this!

The Needs Review Queue Bot → tested this issue. It fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide → to find step-by-step guides for working with issues.

Fixed theretical bug that might occur when we run out of phpstan errors. Unlikely, but possiblee.

Keeping RTBC since the change was so minimal.

Sorry, didnt switch branch for another issue. Reverted

Nice little addition to our GitLab reporting, thanks for adding it.

Committed 718a97e and pushed to 11.x. Thanks!

longwave → closed merge request !10885

Awesome. :)

We perhaps should make a meta issue to see what metrics would be usefull to expose in the different jobs. There is possibly a lot more :)

📌 [PP-1][CI] Spin off Drupal Components tests in a job of their own Postponed adds some more.