Test with 11.2.0-rc2

Thanks for looking at this. I've pushed a change to skip all the other variants and jobs we are not interested in, and also only run two of the parallel streams automatically. The others are now manual. Makes the whole thing quicker.

Thanks @jonathan1055; I appreciate your reporting the issue and support now.

I don't know how common it is for Contrib to use the @group way to select tests, but I think I'm right in that it is a documented way that run-tests.sh can be used. I have started the process of adding the new attribute meta, 📌 Add #[Group] attributes for Core 11.2 and PHPUnit 11 Active but that is taking more effort, because the deprecations are still being shown (at least they did when run against 11.2 before the lastest commits on 📌 Deprecate TestDiscovery test file scanning, use PHPUnit API instead Active )

Well we may argue there are too many ways to filter and get the tests needed for running, but we need to support them all ATM, alas. Simplifying that part would be nice, but I guess that will require deciding how and the deprecating hownots.

For converting annotations to attributes, I worked on a rector rule if you want to try - https://www.drupal.org/project/drupal/issues/3446380 📌 [meta] Define a Rector rule to convert test annotations to attributes Active

There are two issues in NR that were built using that rule, 📌 Convert test annotations to attributes in Drupal/BuildTests Active and 📌 Convert test annotation to attributes in Drupal/Test/Component Active

I could replicate the issue in Image Effects https://git.drupalcode.org/project/image_effects/-/merge_requests/72 by choosing to select tests by group instead of by directory.

To me the issue is a bug upstream: PHPUnit 'builds' the list of tests to execute as a hierarchical structure of TestSuite and TestCase objects, and seems to be adding empty TestSuite sub-objects to the main TestSuite object under some unclear circumstances.

Core patch developed here seems to work, and it is now a follow-up MR in core's #3497431-88: Deprecate TestDiscovery test file scanning, use PHPUnit API instead → .

It just skips, though, empty TestSuite objects which may lead to skipped test runs if the @group/#[Group] of the test class skipped corresponds to the group filter when running runt-tests.sh. I will keep investigating in Image Effects issue 📌 Test with 11.2.0-rc2 Active .

I think core's 📌 Allow indicating alternative phpunit.xml than core's when testing via run-tests.sh Active would help as it could open up the possibility to declare the directories to be searched for test discovery by the module, if run-tests.sh execution is filtered by test group. This will prevent discovering all of the core test files as it is happening now.

Well we may argue there are too many ways to filter and get the tests needed for running, but we need to support them all ATM

Yes, the phpunit binary supports --group mygroup so it is important that we have the same (similar) functionality in run-tests.sh. You can see in the Scheduler .gitlab-ci.yml I have added a bit in before_script to allow swapping between _PHPUNIT_CONCURRENT: 1 and _PHPUNIT_CONCURRENT: 0 without any further changes. This has proved very useful in my Gitlab Tempates development work.

I could replicate the issue in Image Effects ... by choosing to select tests by group instead of by directory.

Exactly. Yes that is one of the documented ways we show how Contrib can use their Gitlab variabls to make customization. So other projects useing that method would also break with this change.

This is a very good investigation and it is great that you have isolated the problem. Regarding core issue 📌 Allow indicating alternative phpunit.xml than core's when testing via run-tests.sh Active we also have 🐛 beStrictAboutOutputDuringTests=true or failOnRisky=true causing test failures due to PHP8.4 Implicit Nullable deprecation Active in Gitlab Templates, which may help with this.

New error must be unrelated
Error when performing the request to https://repo.yarnpkg.com/4.9.1/packages/yarnpkg-cli/bin/yarn.js; for troubleshooting help, see https://github.com/nodejs/corepack#troubleshooting at fetch (/usr/lib/node_modules/corepack/dist/lib/corepack.cjs:22051:11)

I just wanted to check that it works with plain phpunit binary (_phpunit_concurrent=0) and it does
https://git.drupalcode.org/project/scheduler/-/jobs/5563832

So now restoring back to using run-tests.sh (_phpunit_concurrent=1)

So I think I nailed the problem(s). Will post a new patch for core soon then explain.

So. It turns out that PHPUnit is right, and that the error that were reported are legit failures of the discovery process, that was working fine.

In the latest patch, I added a tracer that collects and then reports all events thrown internally by PHPUnit while searching for the test classes to execute.

Here we find now a lot of info, but this is the interesting one:

* PHPUnit\Event\Test\PhpunitErrorTriggered: The data provider specified for Drupal\Tests\Composer\Generator\BuilderTest::testBuilder is invalid
Class "Drupal\Composer\Composer" not found
/builds/issue/scheduler-3530183/web/core/lib/Drupal/Core/Test/PhpUnitTestDiscovery.php:113
/builds/issue/scheduler-3530183/web/core/scripts/run-tests.sh:1049
/builds/issue/scheduler-3530183/web/core/scripts/run-tests.sh:189

* PHPUnit\Event\TestRunner\WarningTriggered: No tests found in class "Drupal\Tests\Composer\Generator\BuilderTest".

* PHPUnit\Event\Test\PhpunitErrorTriggered: The data provider specified for Drupal\Tests\ComposerIntegrationTest::testComposerTilde is invalid
The "/builds/issue/scheduler-3530183/web/composer" directory does not exist.
/builds/issue/scheduler-3530183/web/core/lib/Drupal/Core/Test/PhpUnitTestDiscovery.php:113
/builds/issue/scheduler-3530183/web/core/scripts/run-tests.sh:1049
/builds/issue/scheduler-3530183/web/core/scripts/run-tests.sh:189

* PHPUnit\Event\Test\PhpunitErrorTriggered: The data provider specified for Drupal\Tests\address\Unit\Plugin\Validation\Constraint\CountryConstraintValidatorTest::testValidate is invalid
Data Provider method Drupal\Tests\address\Unit\Plugin\Validation\Constraint\CountryConstraintValidatorTest::providerTestValidate() is not static
/builds/issue/scheduler-3530183/web/core/lib/Drupal/Core/Test/PhpUnitTestDiscovery.php:113
/builds/issue/scheduler-3530183/web/core/scripts/run-tests.sh:1049
/builds/issue/scheduler-3530183/web/core/scripts/run-tests.sh:189

* PHPUnit\Event\TestRunner\WarningTriggered: No tests found in class "Drupal\Tests\address\Unit\Plugin\Validation\Constraint\CountryConstraintValidatorTest".

* PHPUnit\Event\Test\PhpunitErrorTriggered: The data provider specified for Drupal\Tests\address\Kernel\Formatter\AddressPlainFormatterTest::testRender is invalid
Data Provider method Drupal\Tests\address\Kernel\Formatter\AddressPlainFormatterTest::renderDataProvider() is not static
/builds/issue/scheduler-3530183/web/core/lib/Drupal/Core/Test/PhpUnitTestDiscovery.php:113
/builds/issue/scheduler-3530183/web/core/scripts/run-tests.sh:1049
/builds/issue/scheduler-3530183/web/core/scripts/run-tests.sh:189

* PHPUnit\Event\Test\PhpunitErrorTriggered: The data provider specified for Drupal\Tests\token\Kernel\LanguageTest::testLanguageTokenReplacement is invalid
Data Provider method Drupal\Tests\token\Kernel\LanguageTest::languageTokenReplacementDataProvider() is not static
/builds/issue/scheduler-3530183/web/core/lib/Drupal/Core/Test/PhpUnitTestDiscovery.php:113
/builds/issue/scheduler-3530183/web/core/scripts/run-tests.sh:1049
/builds/issue/scheduler-3530183/web/core/scripts/run-tests.sh:189

* PHPUnit\Event\Test\PhpunitErrorTriggered: The data provider specified for Drupal\Tests\token\Kernel\LanguageTest::testCurrentPageLanguageTokenReplacement is invalid
Data Provider method Drupal\Tests\token\Kernel\LanguageTest::currentPageLanguageTokenReplacementDataProvider() is not static
/builds/issue/scheduler-3530183/web/core/lib/Drupal/Core/Test/PhpUnitTestDiscovery.php:113
/builds/issue/scheduler-3530183/web/core/scripts/run-tests.sh:1049
/builds/issue/scheduler-3530183/web/core/scripts/run-tests.sh:189

* PHPUnit\Event\TestRunner\WarningTriggered: No tests found in class "Drupal\Tests\token\Kernel\LanguageTest".

...

* PHPUnit\Event\Test\PhpunitErrorTriggered: The data provider specified for Drupal\Tests\address\FunctionalJavascript\AddressViewsHandlersTest::testAddressFieldHandlers is invalid
Data Provider method Drupal\Tests\address\FunctionalJavascript\AddressViewsHandlersTest::addressFieldHandlersDataProvider() is not static
/builds/issue/scheduler-3530183/web/core/lib/Drupal/Core/Test/PhpUnitTestDiscovery.php:113
/builds/issue/scheduler-3530183/web/core/scripts/run-tests.sh:1049
/builds/issue/scheduler-3530183/web/core/scripts/run-tests.sh:189

* PHPUnit\Event\Test\PhpunitErrorTriggered: The data provider specified for Drupal\BuildTests\Composer\ComposerValidateTest::testValidateComposer is invalid
The "/builds/issue/scheduler-3530183/web/composer" directory does not exist.
/builds/issue/scheduler-3530183/web/core/lib/Drupal/Core/Test/PhpUnitTestDiscovery.php:113
/builds/issue/scheduler-3530183/web/core/scripts/run-tests.sh:1049
/builds/issue/scheduler-3530183/web/core/scripts/run-tests.sh:189

* PHPUnit\Event\TestRunner\WarningTriggered: No tests found in class "Drupal\BuildTests\Composer\ComposerValidateTest".

IMHO, this means that

a) the GitLab Templates is not symlinking core's /composer subdirectory, so the classes cannot be autoloaded. Quickly checked the script that does that there, and in fact it is missing;

b) in the address module, some Data Provider methods are not static. This is a failure in PHPUnit 11;

c) same in the token module.

So issues should be filed against each of those projects for fixing the problems.

run-tests.sh test discovery is not failing, anyway, but it will not include tests that have legitimate problems. And now the causes are reported too so actions can be taken.