"Build Successful" is treated as a test failure

Created on 4 August 2018, over 6 years ago
Updated 30 January 2023, almost 2 years ago

I noticed there has been a recent change to how the testbot reports test results.

Some projects I contribute to have no test cases, but have the testbot turned on anyway. This helps the project maintainer because the testbot will check patches to ensure they apply and are free of coding standards errors. And of course this is a first step towards getting actual test cases committed, so it's not something we want to discourage.

Up until a few weeks ago, when there were no test cases the testbot would report something like "PHP 5.5 & MySQL 5.5, D8.7 No tests found" with a "green" status indicating the patch was successful. The issue status would remain at "Needs review".

Now, however, the testbot is reporting the same patch as "PHP 5.5 & MySQL 5.5, D8.7 Build Successful" which is NOT "green", and thus the testbot sets the issue status to "Needs work". This makes it appear like the test has failed! (See #2938963-3: Sender "Mime Mail mailer" fails without explanation β†’ for an example.)

This first started happening on 18 July 2018 - at https://www.drupal.org/pift-ci-job/1081420 β†’ you can see that on and before 17 July 2018 the testbot reported "No tests found", while on and after 18 July 2018 the testbot reported "Build Successful" and failed the test.

This is a regression from the behavior established in #2645590: Ensure that simpletest job doesn't "fail" testing if no tests are present β†’ , where the testbot was changed so that

projects with no tests will now return a "No tests Found" and be green if the build is successful

Can we go back to treating "Build Successful" as a "green" status? Marking this as a bug because "Build Successful" is not a failure.

πŸ› Bug report
Status

Active

Component

Testrunner Tests

Created by

πŸ‡ΊπŸ‡ΈUnited States tr Cascadia

Live updates comments and jobs are added and updated live.
  • Regression

    It restores functionality that was present in earlier versions.

Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

Production build 0.71.5 2024