The "phpunit (next major)" pipeline considers all files in the module ending with "Test" as test files

Thanks for reporting this. Can you give a link to your pipeline so we can investigate.
It's interesting that the 'current' phpunit job (Drupal 10) runs OK but not the 'next major' (Drupal 11).

This might be then a change with the run-tests.sh core script, so the issue might be there.

A workaround while the issue is worked on is to rename the file to, for example: MyModuleTestBase.php

Sorry for the delay, it's because the module, causing this issue, is not published as opensource yet.

So, I decided to reproduce this issue with my other module - here are the failing pipelines for PHPUnit:
https://git.drupalcode.org/project/test_helpers/-/jobs/3090604
https://git.drupalcode.org/project/test_helpers/-/jobs/3090605

And the change to trigger this issue is pretty simple:
https://git.drupalcode.org/project/test_helpers/-/merge_requests/116/diffs

So, please review it and try to find the source of this issue, cuz I still have no idea why this started to fail on Drupal 11 but still works well on Drupal 10.

Thanks for the links and custom MR. I have just set OPT_IN_PREVIOUS_MAJOR so we can see if this passes at D10, as that was one of the original questions.

Also, there is no gitlab_templates MR so this is 'needs work' not 'needs review' :-)

The standards for naming conventions → say

Test classes should always have the suffix "Test".

but is there an implication that non-test classes should not have the suffix "Test" ?

I did some work on your pipeline, to run "previous major" PHPUnit 9 and compare it to "current" PHPUnit 10 to see the differences
https://git.drupalcode.org/issue/test_helpers-3481744/-/pipelines/314112

For easy reference here is a link to your tests/modules/test_helpers_test/src/TestHelpersTest.php

I think this is probably an enhancement/feature in PHPUnit 10, just tightening up on what it considers should be test classes, given the file naming. It is certainly nothing that gitlab_templates have introduced, other than moving forward to Drupal 11 being considered the 'current' core version to test against, and therefore using PHPUnit 10.

The log does say 'warning' not 'error' so there may be an option to override the behavior and not make the job fail for this?

Wrong status and category, changing to 'active' support request.

For Drupal 10 discovery of the tests is done by the tests/TestSuites/UnitTestSuite.php that has a deprecation comment:

/**
 * Discovers tests for the unit test suite.
 *
 * @deprecated in drupal:10.3.0 and is removed from drupal:11.0.0. There is no
 *   replacement and test discovery will be handled differently in PHPUnit 10.
 *
 * @see https://www.drupal.org/node/3405829
 */

So, with this file only files, having the Drupal\Tests workspace were included into tests.

But from Drupal 11 it is removed, seems this causes the issue.

A workaround to not use "Test" suffixes in file names sounds to me pretty strange, because many modules contain helper submodules in the [my_module]/tests/modules directory (to hide them from installing manually, that is recommended way) and the common name for them is [my_module_test], so the common class name is to follow the submodule name.

So, let's try to implement some fix to not restrict usage the "Test" suffixes for test modules.

Seems we can fix this by extending the PHPUnit’s XML configuration file, documentation is here: https://docs.phpunit.de/en/10.5/organizing-tests.html - so, something like this should help:

        <testsuite name="unit">
            <directory>tests/unit</directory>
        </testsuite>

Note that you can have your own module's phpunit.xml file and that it will be considered when running the tests, so maybe this is the best thing to do here.

Yes I was curious to see how that would work, and tried it but without success. I think it may have worked if the phpunit job executed the tests by using a named suite, but by default the template does not do that. I may be possible with _PHPUNIT_EXTRA. Currently we cater for specifying _PHPUNIT_TESTGROUPS but we do not do anything about named test suites.

The phpunit documentation explicitly says

If you point the PHPUnit command-line test runner to a directory it will look for *Test.php files.

So you could say that this is problem with Drupal's recommend naming convention for test helper modules to end in _test as that is the root cause of this difficulty. If the recommendation (I don't know if it is a written standard) could allow modules that exist for ther beneift of testing to be named _testing then that would solve this problem. I guess it did not surface when there was the class for test discovery in core 10.3 but there may be other testing modules that will have this problem now.

Yes, we surely can invent a local workaround to get rid of this error. Actually, I got this error not with the test_helpers module, but with the module named "commercetools", where we have to create a helper submodule "commercetools_test" with helpers for functional testing.

So, the best solution for now is to find a correct fix to not restrict users to name files with the "Test.php" ending at all. Because anyone in any module (contrib or private) can face this problem and will get confused a lot!

If this is not possible, let's then extend the Drupal documentation with a warning that all modules, placed in the tests/modules directory should not have any files with the "Test.php" ending.

But how many developers read the documentation? ;-)

So, I'll try to find time to find a right solution for this issue.

I don't think altering the definition of the test suite directories is going to help here. I've been testing this in Scheduler MR194 where I have reproduced your error

Class Drupal\scheduler_api_test\SchedulerApiTest declared in /builds/project/scheduler/tests/modules/scheduler_api_test/src/SchedulerApiTest.php does not extend PHPUnit\Framework\TestCase

This error is thrown regardless of which tests are being executed. See this pipeline where I filter for just one kernel test. The scheduler_api_test module is not even needed for that test, no api tests are run but the error is still reported and the job fails. Locally I ran this with debug and it seems that the entire scheduler/tests/ folder is scanned for files ending Test.php (as per the phpunit docs I linked in #13). This is because I ran the tests by giving modules/scheduler as the starting point. See attached local log. It gitlab pipelines we do the same thing, using $_WEB_ROOT/modules/custom/$CI_PROJECT_NAME

I can avoid this error locally by using modules/scheduler/tests/src as the starting point for the tests. This has the downside that it does not detect tests in a Scheduler sub-module, stored in scheduler_rules_integration/tests/src so it's not a valid solution for gitlab_templates to implement, but it demonstrates the problem and a possible customised solution for you to try locally. There is not currently any way to vary that argument in the phpunit execution statement, but we could investigate turning it into a variable which could then be customised.

Another solution is to talk to the Coding Standards team → and find out exactly how and where the naming of test helper modules is documented, and if we can allow an alternative ending, as I suggested above.

Seems we have a solution for this issue in PHPUnit 11.x - here https://github.com/sebastianbergmann/phpunit/pull/5629 is a PR that adds the --exclude-filter option where we probably can pass the excluding pattern like **/src/**.