[meta] Known intermittent, random, and environment-specific test failures

Starting to see some random FunctionalJavascript ones popup. When they fail they sent back over a dozen tickets at once.

https://www.drupal.org/pift-ci-job/2628256 →

Testing Drupal\Tests\media\FunctionalJavascript\MediaSourceFileTest
E                                                                   1 / 1 (100%)

Time: 00:09.719, Memory: 4.00 MB

There was 1 error:

1) Drupal\Tests\media\FunctionalJavascript\MediaSourceFileTest::testMediaFileSource
Behat\Mink\Exception\ElementHtmlException: The text "text--plain.png" was not found in the attribute "src" of the element matching css "img".

https://www.drupal.org/pift-ci-job/2628180 →

Testing Drupal\Tests\layout_builder\FunctionalJavascript\ContextualLinksTest
F                                                                   1 / 1 (100%)

Time: 00:18.127, Memory: 4.00 MB

There was 1 failure:

1) Drupal\Tests\layout_builder\FunctionalJavascript\ContextualLinksTest::testContextualLinks
Failed asserting that a NULL is not empty.

/var/www/html/vendor/phpunit/phpunit/src/Framework/Constraint/Constraint.php:122
/var/www/html/vendor/phpunit/phpunit/src/Framework/Constraint/Constraint.php:55
/var/www/html/core/modules/layout_builder/tests/src/FunctionalJavascript/ContextualLinksTest.php:111
/var/www/html/core/modules/layout_builder/tests/src/FunctionalJavascript/ContextualLinksTest.php:80
/var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php:728

Here's a new one 🐛 [random failure] Curl error thrown for http in JSWebAssertTest Fixed discovered in https://www.drupal.org/pift-ci-job/2629301 →

Same as #166 on this issue ✨ Project messaging channel in core (as experimental) Fixed and this job: https://www.drupal.org/pift-ci-job/2629786 →

Remove fixed issue and add a child to the IS that was not in the list.

There's new InstallerExistingConfigMultilingualTest after upgrade to Symfony 6.3@beta fires very often

Adding 🐛 [random test failure] InstallerExistingConfig[SyncDirectory]MultilingualTest::testConfigSync Fixed

Adding 🐛 Double exposed filter field when trying to use "q" as identifier Fixed .

And now adding :)

🐛 Double exposed filter field when trying to use "q" as identifier Fixed is an issue that *encountered* a random fail, not an issue *describing* one, removing it from the IS.

(That issue was hitting InstallerExistingConfigSyncDirectoryMultilingualTest and InstallerExistingConfigMultilingualTest, which were recently addressed in 📌 Skip InstallerExistingConfig[SyncDirectory]MultilingualTest::testConfigSync Fixed )

Oops, sorry @tim.plunkett, and thanks for fixing it.

I've been getting these lately,
https://www.drupal.org/pift-ci-job/2734245 →

1) Drupal\Tests\system\Functional\System\CronRunTest::testAutomatedCron
Failed asserting that 1691374878 is less than 1691374878.

I have not made a child issue.

Got the error above again so I made the issue and added to the IS.

Removed fixed issues from IS.

I saw a failure in core/modules/media/tests/src/FunctionalJavascript/MediaSourceOEmbedVideoTest.php that went away when I re-ran the (GitLab) pipeline. It looks different from the failure reported in #166 on this issue.

https://git.drupalcode.org/issue/drupal-3386762/-/jobs/245517#L405

    Drupal\Tests\media\FunctionalJavascript\MediaSourceOEmbedVideoTest::testMediaOEmbedVideoSource
    WebDriver\Exception\JavaScriptError: javascript error: Cannot read
    properties of null (reading 'getAttribute')
      (Session info: headless chrome=106.0.5249.103)
      (Driver info: chromedriver=106.0.5249.61
    (511755355844955cd3e264779baf0dd38212a4d0-refs/branch-heads/5249@{#569}),platform=Linux
    5.4.241-150.347.amzn2.x86_64 x86_64)
    
    /builds/issue/drupal-3386762/vendor/instaclick/php-webdriver/lib/WebDriver/Exception.php:198
    /builds/issue/drupal-3386762/vendor/instaclick/php-webdriver/lib/WebDriver/AbstractWebDriver.php:166
    /builds/issue/drupal-3386762/vendor/instaclick/php-webdriver/lib/WebDriver/Session.php:436
    /builds/issue/drupal-3386762/vendor/behat/mink-selenium2-driver/src/Selenium2Driver.php:973
    /builds/issue/drupal-3386762/vendor/behat/mink/src/Session.php:343
    /builds/issue/drupal-3386762/core/modules/media/tests/src/FunctionalJavascript/MediaSourceOEmbedVideoTest.php:147
    /builds/issue/drupal-3386762/vendor/phpunit/phpunit/src/Framework/TestResult.php:728

Updated latest failed date for 🐛 [random test failure] DialogTest::testDialog() Active in the IS>

At least some of the 'new' failures on gitlab seem like they might be a regression of https://git.drupalcode.org/project/drupalci_testbot/-/commit/b5e9ad15bcb...

I opened #3317520: Random failure in Drupal\Tests\settings_tray\FunctionalJavascript\SettingsTrayBlockFormTest::testEditModeEnableDisable → but it's a duplicate of 🐛 [random test failure] SettingsTrayBlockFormTest:: testEditModeEnableDisable() Active which we hadn't seen on DrupalCI since the ipv6 issue was fixed.

New one popped up today 🐛 [random test fail] TimestampFormatterWithTimeDiffViewsTest::testTimestampFormatterWithTimeDiff Active

https://git.drupalcode.org/issue/drupal-3395404/-/pipelines/49731/test_r...

#191: i've been seeing that one for at least a few days, and I'm pretty sure I've been seeing it since Mon Nov 6.

I also got the one in #188 today. A retest fixed it, I did notice a few dead URL's in that test so not sure if that might be related or not. Create an issue for that as well: 🐛 Media MediaSourceOEmbedVideoTest is testing dead links Active

Is it just me or have the random failures with night watch and JavaScript gone up with gitlab? And I love gitlab and hope that’s not the case but something I’ve noticed the last 2 weeks

It's definitely gone up, we're seeing frequent failures come back that were previously fixed on DrupalCI via preventing IPV6 requests, but that doesn't appear to be the problem this time, or if it is it's hard to verify, see #3397749: gitlab pod DNS settings → for background.

However I've revived 📌 Stabilize FunctionalJavascript testing AJAX: add ::assertExpectedAjaxRequest() Fixed which should help/fix essentially all AJAX tests once it's committed or at least help us to fix any remaining ones.

After that the next worst one, and not fixed by the AJAX assertion changes is #3317520: Random failure in Drupal\Tests\settings_tray\FunctionalJavascript\SettingsTrayBlockFormTest::testEditModeEnableDisable → which has at least one method skipped but might need more until we're able to fix the underlying thing for that too.

Re #194 that can often be the case when your CI environment, or site in general, is more performant. That's because things are faster and tests that previously passed because they took long enough to "wait" for things themselves no longer do so assertions can start failing.

None of the DNS changes we did for the settings ("single-request" and "ndots") made a difference, so we might consider reverting them.

We recently increased the resources requested from some jobs to the pods, as that change seems to have an impact.

I've been doing some additional debugging here ✨ [investigation] get metrics of request to see which settings need to be changed for random failures Active , where I'm unskipping most of the skipped tests, and so far I'm only getting one or two repeated offenders, but most of the others no longer fail, or at least I couldn't reproduce.

I'm adding additional artifacts (screenshots and html-shot of the failing step) that might help with debugging, but all that is not finished yet. I just wanted to give an update here.

Added 🐛 mysql\ConnectionUnitTest::testOpenQueryPrefetchClose Active

Drupal\FunctionalJavascriptTests\Ajax\AjaxTest failed for me today, even after 📌 [random test failure] Re-enable AjaxTest::testAjaxFocus() Needs work

ref https://git.drupalcode.org/issue/drupal-3421482/-/jobs/927005

    1) Drupal\FunctionalJavascriptTests\Ajax\AjaxTest::testAjaxFocus
    Failed asserting that two strings are equal.
    --- Expected
    +++ Actual
    @@ @@
    -'edit-textfield-3'
    +'edit-textfield-2'
    
    /builds/issue/drupal-3421482/vendor/phpunit/phpunit/src/Framework/Constraint/Equality/IsEqual.php:94
    /builds/issue/drupal-3421482/core/tests/Drupal/FunctionalJavascriptTests/Ajax/AjaxTest.php:335
    /builds/issue/drupal-3421482/vendor/phpunit/phpunit/src/Framework/TestResult.php:728

The latest revision of the OOP Hook patch I am working on also failed on that AjaxTest. https://git.drupalcode.org/issue/drupal-3442246/-/pipelines/164384

I've only seen this once and noting it here.

    Drupal\Tests\layout_builder\Functional\LayoutBuilderBlocksTest::testBlockPlaceholder
    Behat\Mink\Exception\ResponseTextException: The text "Placeholder for the
    "The block label" block" appears in the text of this page, but it should
    not.

Re #202 I just ran into this and GitLab CI notes that this test "Failed 15 times in 11.x in the last 14 days" so this one probably needs an issue to investigate.

Created 🐛 [random test failure] LayoutBuilderBlocksTest::testBlockPlaceholder failing Active for the testfailure mentioned in #202 and #203.

There are several reports on 11.x for Drupal\Tests\block\Functional\BlockCacheTes but not always the same test

Drupal\Tests\block\Functional\BlockCacheTest::testCachePermissions - 23x in 14 days
Drupal\Tests\block\Functional\BlockCacheTest::testCachePerPage - 14x in 14 days

Add all child issues to the issue summary and remove the one that was fixes.

I see two test failures on the latest pipeline (yesterday) for https://git.drupalcode.org/project/drupal/-/merge_requests/6111/pipeline... (for 📌 Implement Entity::fields() for migration destinations Needs work ):

1. Drupal\Tests\image\Functional\ImageStylesPathAndUrlTest::testImageStyleUrlAndPathPrivate
2. Drupal\Tests\responsive_image\FunctionalJavascript\ResponsiveImageFieldUiTest::testResponsiveImageFormatterUi

These two tests seem unrelated to the issue, and both tests passed when I re-ran them.

Re: #208: Yeah, I'm also seeing a lot of fails for ImageStylesPathAndUrlTest recently. Opened 🐛 [random test failure] ImageStylesPathAndUrlTest Active for it.

Probably needs IS update?
I had to scan all comments. In a single MR (https://git.drupalcode.org/issue/drupal-3493296/-/jobs/3793752) I had failures for:

• 🐛 [random test failure] ImageStylesPathAndUrlTest Active
• 🐛 [random test failure] CronRunTest::testAutomatedCron Fixed

(which apparently was fixed, but doesn't look like)

• 📌 [random test failure] BlockCacheTest Active
• 🐛 [random test failure] LayoutBuilderBlocksTest::testBlockPlaceholder failing Active

In #3498915-14: Move content_entity source plugin to migrate module → , I noted several failing tests:

1. Drupal\Tests\jsonapi\Functional\JsonApiFunctionalTest
2. Drupal\Tests\file\Functional\DownloadTest
3. Drupal\Tests\package_manager\Build\PackageInstallTest
4. Drupal\Tests\package_manager\Build\PackageInstallSubmoduleTest
5. Drupal\Tests\package_manager\Build\PackageUpdateTest
6. tabbingManagerTest.js (I think: certainly one of the Nightwatch tests)

I see there is already a child issue 🐛 [random test failure] EditorSecurityTest::testEditorXssFilterOverride Active , but I do not see the others mentioned here. All these tests passed when I re-ran them.

Added 🐛 [random test failure] PathAliasTest::testPathCache Active

Are there any issues tracking Nightwatch random fails? It seems to fail almost 50% of the time for me across multiple MRs. Here's the latest one https://git.drupalcode.org/issue/drupal-293295/-/jobs/4349289. The other day it was a ckeditor5 failure but the error message was about the database not being reachable.

Heaps of random Nightwatch fails here https://git.drupalcode.org/issue/drupal-2736401/-/pipelines/426966/test_...

Been seeing ConfigTranslationUiModulesTest fails across multiple issues today.

Added 📌 [consistent test failure] ImageDimensionsTest::testImageDimensions() Active as a replacement of #2921123: Adjust Rectangle class to calculate rotated image dimensions according to libgd 2.2.2+ → .

Added 🐛 [random test failure] JsonApiFunctionalTest::testRead Active

Sidenote - should we remove the issue list from the IS that is out of date and just rely on child issues?

Removing fixed issues from the IS.