Ensure run-tests.sh and PHPUnit CLI run with the same list of tests to be executed

Created on 29 September 2024, 5 months ago

Problem/Motivation

When running tests with filtering (e.g. by test suite, or by group), run-tests.sh Drupal own runner and PHPUnit CLI runner should ideally discover the same list of test classes to be executed. This is currently not happening, because:

  1. Some core tests classes do not have a name that ends with *Test. These tests are found by run-tests.sh but not by PHPUnit.
  2. Some core tests classes are placed in directories that PHPUnit 10 cannot reach via the current phpunit.xml.dist configuration. These are tests placed in core submodules. Unfortunately, usage of the globstar pattern like <directory>modules/**/tests/src/Unit</directory>
  3. was only introduced in PHPUnit 11, https://github.com/sebastianbergmann/php-file-iterator/pull/83. Until then, the only possibility is to explicitly indicate the path in phpunit.xml.dist.

Steps to reproduce

Proposed resolution

Fix the issues and add a test that continuosly compares the result of PHPUnit and TestDiscovery test discovery lists and fails if they differ.

Remaining tasks

User interface changes

Introduced terminology

API changes

Data model changes

Release notes snippet

🐛 Bug report
Status

Active

Version

11.0 🔥

Component

phpunit

Created by

🇮🇹Italy mondrake 🇮🇹

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @mondrake
  • Merge request !9684Closes #3477634 → (Closed) created by mondrake
  • Pipeline finished with Success
    5 months ago
    Total: 1138s
    #296232
  • 🇮🇹Italy mondrake 🇮🇹
  • The Needs Review Queue Bot tested this issue. It fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

    This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

    Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

  • 🇮🇹Italy mondrake 🇮🇹

    rebased

  • Pipeline finished with Failed
    5 months ago
    Total: 141s
    #306608
  • Pipeline finished with Failed
    5 months ago
    Total: 136s
    #306623
  • Pipeline finished with Failed
    5 months ago
    Total: 135s
    #306631
  • Pipeline finished with Failed
    5 months ago
    Total: 628s
    #306677
  • Pipeline finished with Failed
    4 months ago
    Total: 66s
    #320605
  • Pipeline finished with Failed
    4 months ago
    Total: 675s
    #320618
  • Pipeline finished with Success
    4 months ago
    Total: 563s
    #320692
  • 🇮🇹Italy mondrake 🇮🇹

    Rebased and added <directory>modules/**/tests/src/Build</directory> to build test suite.

  • Pipeline finished with Failed
    4 months ago
    Total: 753s
    #324356
  • Pipeline finished with Failed
    4 months ago
    Total: 841s
    #326370
  • 🇺🇸United States smustgrave

    How come PhpUnitCliTest is removed?

  • 🇮🇹Italy mondrake 🇮🇹

    #7 it's replaced by PhpUnitTestDiscoveryTest , which is a kernel test that does the same thing (call PHPUnit CLI to get the test list), but needs services to build Drupal's list to compare.

  • Pipeline finished with Success
    3 months ago
    Total: 607s
    #342631
  • 🇳🇱Netherlands daffie

    When I run the Drupal\KernelTests\Core\Test\PhpUnitTestDiscoveryTest on my local machine it fails. THe code dump(array_diff(array_values($internalList), array_values($phpUnitList))); returns:

    array:35 [
      1439 => "Drupal\Tests\config_test\Functional\Rest\ConfigTestJsonAnonTest"
      1440 => "Drupal\Tests\config_test\Functional\Rest\ConfigTestJsonBasicAuthTest"
      1441 => "Drupal\Tests\config_test\Functional\Rest\ConfigTestJsonCookieTest"
      1442 => "Drupal\Tests\config_test\Functional\Rest\ConfigTestXmlAnonTest"
      1443 => "Drupal\Tests\config_test\Functional\Rest\ConfigTestXmlBasicAuthTest"
      1444 => "Drupal\Tests\config_test\Functional\Rest\ConfigTestXmlCookieTest"
      1635 => "Drupal\Tests\demo_umami_content\Functional\DefaultContentFilesAccessTest"
      1636 => "Drupal\Tests\demo_umami_content\Functional\UninstallDefaultContentTest"
      1637 => "Drupal\Tests\drupal_system_listing_compatible_test\Kernel\SystemListingCrossProfileCompatibleTest"
      1660 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestBundleJsonAnonTest"
      1661 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestBundleJsonBasicAuthTest"
      1662 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestBundleJsonCookieTest"
      1663 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestBundleXmlAnonTest"
      1664 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestBundleXmlBasicAuthTest"
      1665 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestBundleXmlCookieTest"
      1666 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestComputedFieldNormalizerTest"
      1667 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestJsonAnonTest"
      1668 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestJsonBasicAuthTest"
      1669 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestJsonCookieTest"
      1670 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestJsonInternalPropertyNormalizerTest"
      1671 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestLabelJsonAnonTest"
      1672 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestLabelJsonBasicAuthTest"
      1673 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestLabelJsonCookieTest"
      1674 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestLabelXmlAnonTest"
      1675 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestLabelXmlBasicAuthTest"
      1676 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestLabelXmlCookieTest"
      1677 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestMapFieldJsonAnonTest"
      1678 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestTextItemNormalizerTest"
      1679 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestXmlAnonTest"
      1680 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestXmlBasicAuthTest"
      1681 => "Drupal\Tests\entity_test\Functional\Rest\EntityTestXmlCookieTest"
      2282 => "Drupal\Tests\layout_builder_expose_all_field_blocks\Functional\GenericTest"
      2654 => "Drupal\Tests\navigation_top_bar\Functional\GenericTest"
      2816 => "Drupal\Tests\package_manager\Build\PackageInstallTest"
      2817 => "Drupal\Tests\package_manager\Build\PackageUpdateTest"
    ]
    

    The code changes in the MR look good to me.

  • 🇮🇹Italy mondrake 🇮🇹

    @daffie have you checked your local phpunit.xml vs the one in the MR?

  • Pipeline finished with Success
    3 months ago
    Total: 748s
    #351898
  • 🇮🇹Italy mondrake 🇮🇹

    rebased, still green.

    @daffie this test will work on a vanilla install; if you have custom/contrib modules, it could very well be that there are tests that PHPUnit cannot discover (see the IS point 2). In any case, could you rather share PHPUnit's output than the dump?

  • 🇳🇱Netherlands daffie

    @daffie have you checked your local phpunit.xml vs the one in the MR?

    Oeps, I did not.

    Now that I did do that the test passes on my local machine.
    All code changes look good to me.
    For me it is RTBC.

    @modrake: Thank you for your help.

  • Pipeline finished with Failed
    3 months ago
    Total: 544s
    #354191
  • Pipeline finished with Success
    3 months ago
    Total: 525s
    #354294
  • Pipeline finished with Failed
    2 months ago
    Total: 650s
    #366925
  • Pipeline finished with Success
    2 months ago
    Total: 1288s
    #367057
  • Pipeline finished with Failed
    2 months ago
    Total: 121s
    #377756
  • Status changed to RTBC 2 months ago
  • 🇮🇹Italy mondrake 🇮🇹

    rebased for baseline conflicts

  • Pipeline finished with Running
    2 months ago
    #377761
    • larowlan committed 69f22495 on 11.x
      Issue #3477634 by mondrake, daffie: Ensure run-tests.sh and PHPUnit CLI...
  • 🇦🇺Australia larowlan 🇦🇺🏝.au GMT+10

    Committed to 11.x, thanks for this - will be great to see these caught automatically from here on.

  • 🇧🇪Belgium wim leers Ghent 🇧🇪🇪🇺

    This has bitten me many times before, so this is awesome to see! 👏🙏

  • Automatically closed - issue fixed for 2 weeks with no activity.

Production build 0.71.5 2024