Allow run-tests.sh to generate coverage reports

Thanks for looking at this. Just bumping a comment as it's been 3 years since the last one. Things have probably changed a bit in that time, but having any sort of code coverage report would be an excellent feature to have, for core and contrib.

Here's a re-roll-ish for 10.1.x. When I run it, phpcov runs out of memory for a very small set of tests.

If anyone wants to pick this up, they're welcome to it. :-) I don't have much time to dedicate to it these days.

Setting to needs review to get a test run, but really the next person to come along is welcome to set it back to needs work.

Moved patch #38 into MR
Added new items to CSpell dictionary

Nice work, thanks @mile23 and @voleger. Three other observations:
a) The dispatcher log seems to have run parts of the assessment twice. Is this intentional, or just that the log has duplicated the output?
b) Can we avoid running phpstan on these tests, as that output is unrelated to the MR
c) How can we actually test this here? Can it be run on a small contrib module, or a small subset of core tests?

Just FYI

B) The phpstan errors. These are actually caused because the phpstan library and phpstan-drupal library are updated in this MR. This seems out of scope here. Also some other libraries seem updated, which seems also out of scope.

Re: @jonathan1055 #43.c:

This issue adds a feature that isn't on any maintainer's roadmap, as far as I can tell.

Once upon a time there was #2624926: Refactor run-tests.sh for Console component. → which would refactor run-tests.sh and make it easier to test the test runner, but that's fallen way behind.

Core maintainers have expressed an interest in having coverage reports, but clearly it's not a priority.

One can always run tests using the phpunit test runner and generate coverage for the parts of code you're interested in. For all of Drupal Core this would take a very long time, but it's doable. It's also doable per-module, for instance, for your contrib module or what-have-you.

The Needs Review Queue Bot → tested this issue. It either no longer applies to Drupal core, or fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

Apart from a re-roll or rebase, this issue may need more work to address feedback in the issue or MR comments. To progress an issue, incorporate this feedback as part of the process of updating the issue. This helps other contributors to know what is outstanding.

Consult the Drupal Contributor Guide → to find step-by-step guides for working with issues.

according to https://php.watch/articles/php-code-coverage-comparison

the fastest driver is pcov and it already works with PHP 8.3 (perf of xdebug 3 vs phpdbg is nearly the same nowadays)

@voleger having now gitlab.ci we can prepare special php-image option to enable pcov for tests and report it to gitlab

As of PhpUnit 10 only pcov and xdebug 3 are supported 🌱 [meta] Support PHPUnit 10 in Drupal 11 Active

seems like with gitlab we could start moving toward just using phpunit directly? it would make this pretty trivial and could better integrate with all the other features gitlab provides

I'm trying to set up coverage in https://git.drupalcode.org/project/distributions_recipes/-/merge_request... for a fork of core for an initiative. Ideally to use gitlab's built in coverage I'd be able to add --coverage-text --colors=never to our phpunit command and coverage: '/^\s*Lines:\s*\d+.\d+\%/' to the phpunit gitlab job but PHPUnit is unable to detect a code coverage tool - I guess looking at https://php.watch/articles/php-code-coverage-comparison#phpdbg maybe we can just run it using phpdbg... will see.

I think it will be easier to add disabled by default pcov to all images so it can be enabled with

Will work to provide -dev images to unblock testing #3404084: Add PCOV extension to images →

PHP 8.3 images now has pcov and tests showed that its usage increasing execution time ~30%

The CR https://www.drupal.org/node/3442363 →

So now core can explore usage

8.1 and 8.2 images also updated to have pcov and xdebug preinstalled but not enabled

I suspect the Xdebug problem was just it running in the wrong mode. We could probably add the environment variable or set the default config to always run in coverage mode. I couldn't find the failed test though to confirm this.

Is there a advantage to adding pcov? Xdebug has always worked well enough for me.

@neclimdul you can see changes and pipelines https://git.drupalcode.org/project/distributions_recipes/-/merge_request...

basically

    - docker-php-ext-enable xdebug
    - XDEBUG_MODE=coverage sudo --preserve-env=XDEBUG_MODE,SIMPLETEST_DB,SIMPLETEST_BASE_URL,BROWSERTEST_OUTPUT_DIRECTORY,BROWSERTEST_OUTPUT_FILE,MINK_DRIVER_ARGS_WEBDRIVER -u www-data vendor/bin/phpunit -dzend_extension=xdebug -dxdebug.mode=coverage -v -c phpunit.xml.dist --coverage-text --colors=never --log-junit junit-reports/recipe-tests.xml

Yeah, that looks right. I was suggesting we just add that environment variable to the default container config or the CI config since its unlikely anyone would want any other xdebug mode in CI.

on the php directives, looks like you where passing '-dzend_extension=xdebug -dxdebug.mode=coverage' to phpunit script but you'll need to pass them to php like php -dzend_extension=xdebug -dxdebug.mode=coverage ./vendor/bin/phpunit.

Somehow test suite starting to fail (good example phpdbg -qrr ...phpunit) when PHP executable pointed

Moreover XDebug suggests to use filters for coverage and it should be faster as promoted

You can't run the entire test suite with coverage, it will definitely die. When I ran coverage reports every commit during the 8.x release cycle, I'd only run the unit tests because it doesn't really make sense to run the other tests. They don't provide @covers, and they don't cover specific pieces of code but features. Also, because like you'd have a million tests populating "coverage" for the bootstrap but none of them testing anything.

We actually do the filtering in core its just been renamed in xdebug 3 for obvious reasons.

  <coverage>
    <include>
      <directory>./includes</directory>
      <directory>./lib</directory>
      <directory>./modules</directory>
      <directory>../modules</directory>
      <directory>../sites</directory>
    </include>
    <exclude>
      <directory>./modules/*/src/Tests</directory>
      <directory>./modules/*/tests</directory>
      <directory>../modules/*/src/Tests</directory>
      <directory>../modules/*/tests</directory>
      <directory>../modules/*/*/src/Tests</directory>
      <directory>../modules/*/*/tests</directory>
      <directory suffix=".api.php">./lib/**</directory>
      <directory suffix=".api.php">./modules/**</directory>
      <directory suffix=".api.php">../modules/**</directory>
    </exclude>
</coverage>

Aside from all that, in general I wouldn't surprised if even unit tests for core had problems. I stopped running coverage reports because they'd constantly get broken because core wasn't running them so they don't get reviews and commits would constantly break them in weird ways. That's why I'm here! :-D

xdebug will die on large codebases. pcov won't. try running the whole suite with pcov and compare the coverage scope and run speed (should be 10x faster than xdebug)

Just goning to add that Sebastian Bergmann, author of PHPUnit, is now recommending xdebug over pcov.

https://thephp.cc/articles/pcov-or-xdebug

Spoilers:

After five years of PCOV, I am returning to Xdebug. There are two reasons for this: the configuration setting xdebug.mode introduced with Xdebug 3 and the currently uncertain future of PCOV.

I've no idea how xdebug can be used as it can't even complete running, so promoting it over pcov sounds a nonsense

Part of the reason run-tests.sh is worth keeping around is that it can generate a coverage report per process (thus per test), and then this issue would add the ability to combine all those coverage reports.

See for instance the simpletest_script_generate_clover() function in the patch in #3.