[random test failure] MenuUiTest::testMenuAdministration

Crude should-fail patch that forces the last two menu links are 'z41o8b7c' and 'z5bwlcnj'.

This shows that ['z41o8b7c', 'z5bwlcnj']:
- Sorted using sort(): ['z41o8b7c', 'z5bwlcnj']
- Sorted using \Drupal\Core\Config\Entity\ConfigEntityBase::sort(): ['z5bwlcnj', 'z41o8b7c']

Crude patch to show that if we use \Drupal\Core\Config\Entity\ConfigEntityBase::sort() instead of the default PHP sort(), we're actually testing against the correct expected sorting (whether this is the actual _correct_ sorting is very much up for debate...)

Well, that was the interesting bit, time for the grunt work.

Here's a patch that runs \Drupal\Tests\menu_ui\Functional\MenuUiTest::testMenuAdministration, and only the bit of that test that fails, 2500 times.

And run the 2500x patch together with a 2500x fix-patch.

Hmmm, excuse me whilst I return to the drawing board...

So the problem seem to be that currently

1. The menus, being config entities, are sorted using a "natural order" algorithm (strnatcasecmp).
2. MenuUiTest::testMenuAdministration currently tests the order using sort().

We fixed 2. by using the same sort in the test as used by \Drupal\Core\Config\Entity\ConfigEntityBase::sort.

That solves our woes about testing the last link on the paged list.

However we now still have problems with the first link on the paged list, which seems to point out that the list of menus are not sorted using a "natural order" algorithm, meaning we have failures when the links in the 49th place (last position of the first page) and 50th place (first position on the second page, which we're testing) contain 1 number and multiple numbers.

So for example 's8apdikd' and 's57l1jpq' on the above positions will fail, since the natural sort will put 's8apdikd' after 's57l1jpq', whilst the non-natural sorting of the pager will put 's8apdikd' before 's57l1jpq'.

So after fixing both problems in the test, let's give this another whirl.

Right, MenuUiTest changed recently, just running a new 2500x patch for that to show it still fails.

So that's 6 x 2500 = 15.000 failure free runs with the fixes vs. 8 failures in a 2500 times run of the original test.

Good enough for me.

Actual fix is in the MR.

So that's 6 x 2500 = 15.000 failure free runs with the fixes vs. 8 failures in a 2500 times run of the original test.

Thank you for keeping up with these! those numbers look better to me for sure!

Committed/pushed to 11.x, thanks!

Spokje → closed merge request !4806

Automatically closed - issue fixed for 2 weeks with no activity.