Random failure in Drupal\Tests\settings_tray\FunctionalJavascript\SettingsTrayBlockFormTest::testEditModeEnableDisable

This has resurfaced, and this issue made me wonder whether https://git.drupalcode.org/project/drupalci_testbot/-/commit/b5e9ad15bcb... has regressed on gitlab which would then explain a lot of stability issues. However neither me nor @fjgarlin is sure exactly what the equivalent is on gitlab. I'd opened 🐛 [random test failure] SettingsTrayBlockFormTest:: testEditModeEnableDisable() Active which was a duplicate before rediscovering this.

https://git.drupalcode.org/project/drupal/-/jobs/259057 (1st November 2023).

Postponed on 📌 Skip SettingsTrayBlockFormTest::testEditModeEnableDisable due to random fails Needs review

Adding what this is postponed on to the remaining tasks per Remining tasks → .

This is explicitly not fixed by 📌 Stabilize FunctionalJavascript testing AJAX: add ::assertExpectedAjaxRequest() Fixed so we should revive it.
I'm also seeing

 Drupal\Tests\settings_tray\FunctionalJavascript\SettingsTrayBlockFormTest::testBlocks
    Failed asserting that a NULL is not empty.

Green again.

The Needs Review Queue Bot → tested this issue.

While you are making the above changes, we recommend that you convert this patch to a merge request → . Merge requests are preferred over patches. Be sure to hide the old patch files as well. (Converting an issue to a merge request without other contributions to the issue will not receive credit.)

Updating the title per Special issue titles → .

I tried a few things, results below:

• Unskipping ::testEditModeEnableDisable and repeating the class 1000 times resulted in a fail approx 1 in 3 times (skipped after 38 tests) - https://git.drupalcode.org/issue/drupal-3317520/-/jobs/4819032
• Skipping ::testBlocks and ::testValidationMessages and repeating the class 100 times resulted in 0 fails - https://git.drupalcode.org/issue/drupal-3317520/-/jobs/4819193
• As above, but with usleep removed, 0 fails - https://git.drupalcode.org/issue/drupal-3317520/-/jobs/4819250
• Moving ::testEditModeEnableDisable to its own class and repeating the original class, 0 fails - https://git.drupalcode.org/issue/drupal-3317520/-/jobs/4819383
• As above, but repeating the new test class, 0 fails - https://git.drupalcode.org/issue/drupal-3317520/-/jobs/4819468

So it seems if we simply split ::testEditModeEnableDisable to its own class we can unskip it, but also appears we might be able to remove the usleep. I don't have an explanation for why that might be the case though, I wonder if there's something strange going on with switching themes and if WaitTerminateTestTrait would help.

Using WaitTerminateTestTrait seems to allow us to unskip ::testEditModeEnableDisable without splitting to a separate class - https://git.drupalcode.org/issue/drupal-3317520/-/jobs/4819919

But it failed once where I removed the usleep. Instead of usleep, why don't we just use waitForElementVisible? This seems to pass 100/100 times - https://git.drupalcode.org/issue/drupal-3317520/-/jobs/4819970. And once more for good measure https://git.drupalcode.org/issue/drupal-3317520/-/jobs/4819989.

Have created another MR with just these fixes and updated the issue summary.

mstrelan → changed the visibility of the branch 3317520-settings-tray-repeat to hidden.

I repeated the original job that failed 1 in 3 times and now it's not failing at all. https://git.drupalcode.org/issue/drupal-3317520/-/jobs/4820216

Not sure what to make of that, maybe we don't need the trait after all. But I think the waitForElementVisible makes sense.

Just ran into this in https://git.drupalcode.org/issue/drupal-3482699/-/jobs/4841663, same failure as #15 which is actually a path through several TestBase classes finally landing in OffCanvasTestTrait::waitForOffCanvasArea.

It's hard to tell if the usleep swap would fix this particular case since you'd think it'd fail earlier when calling mouseOver or click on the $block (because find would return NULL?), but it definitely makes sense regardless.

Committed/pushed to 11.x, thanks! Let's see how this one goes.

catch → closed merge request !11684

I'm seeing this fail again with 'stale element not found' as in https://git.drupalcode.org/project/drupal/-/jobs/4929263, reverting for now.

This is pretty hard to debug without knowing what $block_selector, $block_plugin or $new_page_text is. Either way, the block selector element is becoming visible (we're asserting that) but disappearing by the time we try to click it. But if sleeping for 100ms fixes it then I guess the element is eventually being replaced again. Maybe there is something real we can fix to prevent that, but otherwise we probably just need to wait until the element is really ready to be clicked. We could put the whole element find / mouseover / click in to a waitFor callback as a bandaid.