Improve phpunit information in log

a) there is a list as long as the pipeline completes - https://git.drupalcode.org/project/akamai/-/pipelines/11333/test_report.

b) yes, would be helpful. I quickly tried to add --testdox to phpunit call but it didn't work on gitlab ci. It does work on CircleCI, and looks great IMO. I didn't try very hard so so I likely missed something simple.

For (a) thanks for the tip. That's perfect, now I know that the 'test' tab click through to phpunit shows the actual tests. Do we just assume that new users find this? Or is there some gitlab help that I should have read before, to learn this?

The list of tests being run confirmed to me that all tests in all groups were being run even though I had split them into the four test groups using the _PHPUNIT_EXTRA: matrix. This was odd - four streams appear, each labelled with a different group, all failed on the same job. The list of test classes run in the four streams is the same on each. I then realised that I had not changed the source file for the Drush test (this is on the 8.x-1.x branch, I had only done in on the 2.x branch). Thus there were no tests defined for one the four groups in the matrix. This had the side effect that no tests were filtered and all four streams ran all tests. Is this OK, is it what you would have expected? It means if you delete or change an existing group the steam/matrix feature gets scuppered. Just recording this here in case we want to investigate later. The job was https://git.drupalcode.org/project/scheduler/-/pipelines/11422/test_report - all phpunit jobs had 2 skipped, 2 failed and 73 passed.

Thus there were no tests defined for one
the four groups in the matrix. This had the side effect that no tests were
filtered and all four streams ran all tests. Is this OK, is it what you would
have expected? 

Thats surprising to me. I dont think thats an issue for gitlab_templates though. gitlab_template and gitlabci are just responsible for splitting the job into sub-jobs and running the right variant (e.g. appending --group-foo). Test selection is up to `phpunit`. Hope that make sense.

Yes I'm not surprised you say it is a problem out of gitlab_templates scope. I have confirmed what I deduced, by making the change to add the missing group, then re-run the branch pipeline. The four separate matrix jobs ran different subset of tests
https://git.drupalcode.org/project/scheduler/-/pipelines/11424/test_report
The totals do not quite add up to the same as before, we now get 2 failed, 2 skipped but only 71 passes not 73. Making a total of 75 not 77.

Do you think this is worth raising as an issue on the GitLab platform? Not sure exactly where I would do that. Also I don't have the capacity/time at the moment to research this properly, so may leave it til later.

I have checked again the problem discussed in #3-5 above, and I cannot replicate it now. I looked back at the log of the original failing test (where all test groups were run in each of four phpunit sub-jobs. The value of _PHPUNIT_EXTRA was blank in those four phpunit jobs, which was odd, given that this is how we define the job to be split into parallel runs.

However, I have just tried again, with an intentional typo in the --group, and it now works as expected. The incorrect value is passed to _PHPUNIT_EXTRA and no tests are matched for that job (and it passes).

So this problem has either been fixed or my test job had some other error which caused _PHPUNIT_EXTRA to get blanked after the parallel jobs had been created. Either way it is not an issue any more.

Leaving this issue open in case there is anything to do for (b) to reduce the log output length.

I've been checking the one-dot-per-line on the phpunit and it is very weird.
I've even passed the --columns 80 here (https://git.drupalcode.org/issue/api-3420775/-/jobs/794776) and there is no change.

It appears that the --testdox option mentioned in #2 works. See this: https://git.drupalcode.org/issue/api-3420775/-/jobs/798234

If desired, we could use that option and repurpose this issue.

If we want to go the testdox route, the MR is ready: https://git.drupalcode.org/project/gitlab_templates/-/merge_requests/127...

Note that this option will only be valid for "phpunit" (_PHPUNIT_CONCURRENT=0) as "run-tests.sh" does its own output formatting.

This is great. There are three specific benefits that adding --testdox gives us:

• We do not get a dot with a blank line for each test
• Instead we do get the name of the specific tests including the # data sets provided, with a tick or cross to show the result
• The names of each generated ouptut html file are not written to the log. This used to make the log unnecessarily long to scroll through

Example jobs before
https://git.drupalcode.org/project/scheduler/-/jobs/823050

and using MR127 Testdox
https://git.drupalcode.org/project/scheduler/-/jobs/814540

On a short test (Scheduler API group) with 36 tests, 897 assertions, the log reduced by 154 lines from 598 to down to 444 lines, that's a 25% reduction.

On a big test (main Scheduler group) with 180 tests, 4132 assertions, the log size reduced by 1815 lines from 2443 down to 628 lines, a massive 74% reduction.

It may have been the time of day, or other resource/workload factors, but each of my test jobs with --testdox ran in a shorter time than before. But that is on a sample of one, so may not be significant.

This is RTBC!

The briefer of those links is missing its artifacts even when using the download/browse button. Thats not by design, right? It sounded like you expected that only in the log output.

Do you mean the message in the log

WARNING: /builds/project/scheduler/web/sites/default/files/simpletest/phpunit-*.xml: no matching files. Ensure that the artifact path is relative to the working directory (/builds/project/scheduler) 

in https://git.drupalcode.org/project/scheduler/-/jobs/814540 ? That folder sites/default/files/simpletest is always empty, it is like that in the 'before' test.

BUT I had not noticed this, the download of artifacts no longer has the generated test .html files. They should be in sites/simpletest/browser_output, like they are in the 'before' test. Was that what you were meaning?

Putting this back to NW to have that discussion. We need to see if it is possible to have --testdox and to generate the .html files in the same run.

Correct, I refer to the output files not being written and thus not being browseable/downloadable.

So, it seems like the --printer="\Drupal\Tests\Listeners\HtmlOutputPrinter" and --testdox conflict with one another.

Perhaps we can create a variable _PHPUNIT_OUTPUT which defaults to --printer="\Drupal\Tests\Listeners\HtmlOutputPrinter", and then we can add in the documentation page for PHPUnit (https://project.pages.drupalcode.org/gitlab_templates/jobs/phpunit/#usin...) that the variable could also have --testdox for nicer terminal output but no output files.

I went ahead and just added the option to the documentation and added the note about the conflict.
Please review.

I left a couple of comments in the review. Also do we need to highlight the following conflict?

• For SYMFONY_DEPRECATIONS_HELPER: "ignoreFile=$CI_PROJECT_DIR/$_WEB_ROOT/core/.deprecation-ignore.txt" to show deprecations you need _PHPUNIT_CONCURRENT: 1
• But to use --testdox you need _PHPUNIT_CONCURRENT: 0

Therefore you can never have both in the same job.

I think I addressed the comments.

I think that SYMFONY_DEPRECATIONS_HELPER works for both actually. _PHPUNIT_CONCURRENT: 0 uses phpunit, and this uses symfony/phpunit-bridge, which accepts this variable: https://symfony.com/doc/current/components/phpunit_bridge.html#configura...

The --testdox option was made clearer that it is only for _PHPUNIT_CONCURRENT: 0

My mistake, you are right, SYMFONY_DEPRECATIONS_HELPER works for both concurrent = 0 and 1. I had a comment that it did not, and even forced concurrent=1 in my .gitlab-ci.yml before_script when using SYMFONY_DEPRECATIONS_HELPER for 'next minor'. But I just tested and it works for both. Sorry for the noise.

Updated docs page is https://git.drupalcode.org/issue/gitlab_templates-3359927/-/blob/3359927... and looks good. RTBC

Pushed two changes, as agreed on Slack. Still RTBC from me.

Great, I merged these changes now. Thanks for reviewing.

Automatically closed - issue fixed for 2 weeks with no activity.

Corrected the issue summary - changing PHPINFO to PHPUnit