Test-only job does not detect failures correctly

Created on 19 December 2023, 7 months ago
Updated 26 June 2024, 8 days ago

Problem/Motivation

Test-only job sometimes report as successfull even if there are failures in tests, see:

https://git.drupalcode.org/project/drupal/-/jobs/506479

There were 4 errors:
1) Drupal\Tests\system\Functional\Pager\PagerTest::testActiveClass
Error: Class "Drupal\Core\Cache\MemoryCache\MemoryCacheFactory" not found
...
There was 1 failure:
1) Drupal\Tests\user\Kernel\UserPassRehashTest::testUniqueHashNoPasswordValue
...

And still:

Job succeeded

From the Slack discussion with @fjgarlin:

test-only tests are run using phpunit, whereas the whole suite uses run-tests.sh, so that might explain why phpunit is not catching the exception.we have some try/catch blocks on run-tests.sh and on some test base classes that might capture that and not reported

Steps to reproduce

See: https://git.drupalcode.org/project/drupal/-/jobs/506479

Proposed resolution

Fix the error checking.

Remaining tasks

User interface changes

API changes

Data model changes

Release notes snippet

๐Ÿ› Bug report
Status

Fixed

Version

10.3 โœจ

Component
PHPUnitย  โ†’

Last updated about 4 hours ago

Created by

๐Ÿ‡ธ๐Ÿ‡ฐSlovakia poker10

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

Production build 0.69.0 2024