Enable W3C-compliant webdriver testing

alexpott → credited justafish → .

Crediting @justafish as a tonne of the effort here is theirs.

Can you rerun the javascript tests? Want to make sure they aren't random but even I request access I can't rerun.

@smustgrave it's definitely not random.

Glad I asked!

I've started using/recommending this locally too fwiw - a lot of people (more than 2-3 I could name) have trouble with the drupalci one which they don't run into with the selenium one.

I've split the part about upgrading the tests to https://www.drupal.org/project/drupal/issues/3432099 📌 Update our tests to be W3C compatible Active as we can test this is working using geckodriver

After 📌 Use lullabot/mink-selenium2-driver and lullabot/php-webdriver for functional browser testing Fixed I wasn't able run test locally with selenium/standalone-chrome, which I'd otherwise been using successfully for months, gave up trying to debug it and went back to the drupalci image (which has been working OK). I haven't seen anyone else run into issues since that commit, but presumably people will start to switch over if we switch gitlab, so just making a note here in case other people do indeed run into issues down the line.

Just to chime in re #12 I was seeing exactly the same thing (see https://github.com/ddev/ddev-selenium-standalone-chrome/issues/41). It appears that \WebDriver\WebDriver::session() is checking for the w3c flag one level higher up than where it's being set. I'm not sure if the issue should be fixed in lullabot/php-webdriver or by changing the capabilities we're passing. Locally I've just changed the capabilities, but I've seen plenty of examples on Stack Exchange etc. of the w3c flag being within goog:chromeOptions and I can't find a canonical reference on the matter.

I was seeing the same problem as #12, running test on my local machine with Drupal 11.x-dev.

I fixed it by implementing the workaround posted in #13 - I simply replaced line 15 in config.selenium-standalone-chrome.yaml to this:

- MINK_DRIVER_ARGS_WEBDRIVER=[\"chrome\", {\"browserName\":\"chrome\",\"w3c\":false,\"goog:chromeOptions\":{\"args\":[\"--disable-gpu\",\"--headless\", \"--no-sandbox\", \"--disable-dev-shm-usage\"]}}, \"http://selenium-chrome:4444/wd/hub\"]

Thank you @AndyF!

The change record needs to be written but otherwise the tests are green so marking NR, hopefully someone else familiar with Chrome testing can take a look.

My plan was to run a limited subset tests using non-W3C mode. while we still support it - i.e. until Drupal 12 and then in Drupal 12 we should only support W3C testing.

I think we should also add a manual job to run tests using Firefox.

I think we should also add a manual job to run tests using Firefox.

nice idea, there's selenium/standalone-firefox image

alexpott → changed the visibility of the branch 3421202-use-seleniumstandalone-chrome-instead to hidden.

Locally in docker I got few tests running fine using Firefox image with following arguments
MINK_DRIVER_ARGS_WEBDRIVER: '["firefox", {"browserName":"fixrefox","moz:firefoxOptions":{"args":["--headless"]}}, "http://browser:4444"]'

Fixed up issue summary and title to be inline with current MR. Also added more to remaining tasks for things that we need to decide.

Somehow FIrefox tests are x2 faster then selenium jobs

@andypost that's not the case... they were skipping every test :D we should fail instead of skip when a WebDriverTestBase test fails to connect to a browser. Once I fixed the webdriver json string we can see that the tests are about the same speed - see https://git.drupalcode.org/project/drupal/-/pipelines/227038

Also Firefox has a different approach when elements are not in the viewport and you do a click - see https://stackoverflow.com/questions/44777053/selenium-movetargetoutofbou... - I think we should address this in our webdriver in the same way that webdriver-classic-driver does - see https://github.com/minkphp/webdriver-classic-driver/blob/73ad0b6ce21cf69...

Lolz previous me pointed out that PHPUnit would skip if something required by the test is not available - see #3187577-11: FunctionalJavascript tests should fail when ChromeDriver is not running → - I guess that comment is still correct. But it would be great if somehow the reason things are skipped was actually listed in the test output.

To be fair you can see the skipped tests in the report on gitlab - see https://git.drupalcode.org/project/drupal/-/pipelines/227006/test_report

Thanks! it explains difference (all skipped for ff)

btw performance metrics also could be collected from FF, at least it has tracing out of box and allows to enable dev-tools (even via remote somehow) and get better details... for follow-up

Moreover it makes sense to explore optimizations related to --headless mode in browsers

alexpott → changed the visibility of the branch 3421202-on-top-of-3240792 to hidden.

Answered questions in the issue summary and update MR to reflect the answers.

Created the follow-ups.

I think this is ready for review.

One thing for reviewers to not is the move from 2 runners to 4 runners for the JS testing job. This is because testing on selenium is slower. There is not much we can do about this as far as I can see and the benefits of actually testing against current browsers and not maintaining our own chromedriver image are large.

An interesting thing with the four vs. two javascript runners is that overall time still seems to be constrained by the slowest individual tests (either Drupal\Tests\ckeditor5\FunctionalJavascript\MediaTest or Drupal\Tests\ckeditor5\FunctionalJavascript\SourceEditingTest seem to be the longest running) and overall run duration is also similar to what it is now. I would kind of have expected the bottleneck to shift to the number of http requests away from installing Drupal, although maybe those tests also do loads of http requests too.

I wonder if we're hitting concurrency/cpu limits. However anything like that we can explore in a follow-up - four runners seems fine and there's no overall regression in the pipeline feedback.

If we're just moving Nightwatch entirely to w3c mode then we can drop the separate selenium environment and just turn on the w3c switch in the default environment.

Added some nits/comments to the MR.

I've run core/modules/big_pipe/tests/src/FunctionalJavascript/BigPipeRegressionTest.php --filter testMultipleReplacements on both old chromedriver and selenium and the test fails with 2000 and the fix from 🐛 big_pipe sometimes fails to load blocks Active reverted so reducing to 2000 seems fine and we don't need a comment. That was useful to explain why we have reduced it but will be meaningless in the future.

I've answered the rest of @longwave's feedback on the MR - but back to needs review. I think we need to add a nightwatch section to the MR.

Let's update the CR for Nightwatch and also add a followup to investigate Selenium switches, otherwise this is ready to go.

Added testing performance follow-up as requested.

I've tweaked the way W3C mode is set in nightwatch to negate the need for an extra environment. I realised that the selenium environment was missing the merge of chrome args etc... and changing how this is set up seemed best.

I've also update the Cr to reflect the test configuration changes someone would need to make to use W3C compliant browser testing.

Alright, let's ship this - tests are green, and the test and configuration changes are the minimum required to enable W3C mode. This then unlocks some future improvements including hopefully being able to upgrade Nightwatch to a more recent version.

Committed/pushed to 11.x, thanks!

If this does indeed make functional js tests a lot slower, it may break the performance tests dashboard. I've opened 📌 Consolidate Umami performance tests Fixed which should cover that.

Alongside that, I also opened 📌 Consolidate ckeditor5's FunctionalJavascript tests Needs review for core's other longest running functional js tests.

I think we should open a follow-up to revisit concurrency once those two issues are in, since they may be enough to lower it again, I'll open it postponed now.

I am assuming that we don't want to try to backport this to 10.4.x, but if we do, please re-open with a backport MR.

catch → closed merge request !8735

I think we do want to get this in 10.4.x and maybe even get the test base class changes into 10.3.x and 11.x so that projects can move to W3C driver testing early. I'll open when I get round to making the MR for 10.4.x

Performance test job is broken since this went in 🐛 Performance test gitlab job is broken Active .

Created two MRs - one for 11.0.x and one for 10.4.x (and 10.3.x) so we can backport this so that projects can start the process of testing on modern browsers...

The only question is deprecation message, let's update it to "deprecated in 10.0.4 and removed in 12.0.0"

@andypost - ah the best thing to do here is to remove the deprecation... makes core/tests/Drupal/Tests/Core/Test/PhpUnitCliTest.php pass on 10.x without modification which is nice.

Added some comments about being slightly more conservative in the 11.0/10.4 backports. Should this also go back to 10.3 to keep in sync?

It'd be great if this went into 10.3 too.

Great review point @longwave - I've addressed them on both 10.4.x and 11.0.x branches. The 10.4.x also works on 10.3.x

Various tests are broken.

Superfluous dump() doh!

📌 Fix selenium performance/stampede issues in gitlab config and BrowserTestBase Needs review landed so that change needs to be incorporated here now.

Unit tests also failed on 10.4.x in WebAssertTest.

What's the status of this? #47 landed, but then this issue was reopened, without a revert. The two change records are still .

Can contrib follow core's example/lead? Should it?

@Wim This was re-opened for backport, not due to regressions.

However 📌 Fix selenium performance/stampede issues in gitlab config and BrowserTestBase Needs review landed in 11.x, and needs to be incorporated into the backport here to avoid serious performance/instability issues on pipelines. The gitlab YAML files have diverged quite a lot between branches so it is a real backport not just a cherry-pick unfortunately.

Contrib w3c testing is happening in 📌 Update templates so 11.0 is the default/current branch RTBC .

chr.fritsch → made their first commit to this issue’s fork.