Consider dropping Nightwatch in favor of Functional Javascript tests

The thing that this gives us that functional javascript testing doesn't is accessibility testing 📌 Automated A11y tests in Nightwatch Fixed .

However I think this is worth discussing e.g.

1. Is it useful over functional javascript testing for anything other than accessibility testing? If not, should we convert the non-accessibility tests over so that we have minimal nightwatch tests to maintain.

2. Should we try to document better what it's useful for.

In that case it could be worth investigating 📌 Automated A11y tests in PHPUnit Needs work , or seeing if we can use axe-core directly, e.g. via @axe-core/cli or @axe-core/webdriverjs. There is also axe integration with cypress and playwright but I think that's just replacing nightwatch with another similar framework.

I hadn't seen 📌 Automated A11y tests in PHPUnit Needs work but yeah that makes a lot of sense - we could then transfer the nightwatch tests directly over to those.

I've been there ( #3413665-21: Enable modules through Nightwatch API when not testing module enabling → ), but at that time alex pott said:

Re removing Nightwatch - it is doing somethings our PHP based JS testing framework cannot do.

usability testing use Axe - see core/tests/Drupal/Nightwatch/Tests/a11yTestDefault.js
JS unit testing - see core/modules/ckeditor5/tests/src/Nightwatch/Tests/drupalHtmlBuilderTest.js

#3413665-27: Enable modules through Nightwatch API when not testing module enabling →

Thanks @Spokje! We've covered axe in previous comments, but the unit testing bits look like something more suited to jest. The example you provided has a nasty workaround in it already.

General +1 for this and moving unit style JS testing to a more suitable framework (such as jest).

I'm assuming that this is what indicates to Nightwatch that it is a unit test:

'@unitTest': true

There is exactly one occurrence of that in core.

Giving up on that MR, since configuring jest and yarn etc is not my strong suit. But this commit shows what the same test would look like as a jest test.

This is getting a bit off-topic though. Let's focus on the more common usage of Nightwatch.

Worth adding that experience builder is using cypress for testing. No idea why it's using that instead of nightwatch but assuming xb eventually moves to core we definitely shouldn't have all three of functional js, cypress, and nightwatch.

📌 [PP-4] CI: Cypress job should be split in unit vs end-to-end tests Postponed

As expected +1 for dropping Nightwatch and doing axe-stuff elsewhere. I'm convinced all non-JS unit testing can be done (better) in Functional JS Tests.

Worth adding that experience builder is using cypress for testing
[...]
we definitely shouldn't have all three of functional js, cypress, and nightwatch.

Three is definitely a crowd here...
Would be nice to know and maybe even helpful for this issue to known why cypress is chosen.

One of the idea of using nightwatch was to make test writing easier for frontend folks, since it's all JS and you don't need to know a somewhat custom PHP testing tool. Happened to me a couple of times that I was happier writing a nightwatch test instead of having to go for a functionalJS one.

I'm sure we can port most if not all nightwatch tests to functional js tests but at what cost. How would something like toolbarTest.js or toolbarApiTest.js look like? Without even talking about the effort it'd take to convert existing tests instead of working on other things.

Worth adding that experience builder is using cypress for testing. No idea why it's using that instead of nightwatch

Cypress has been really nice in terms of being able to execute the test suite through a UI (not just CLI), and then be able to navigate the browser state at the point of test failure and the stack of states that preceded that failure. Not sure if Nightwatch or other JS testing tools have a similar capability.

We also now have running Cypress (from both CLI and UI) working within DDEV thanks to 📌 DDEV support for Cypress tests Fixed .

OK changing the title here, looks like we have a few options for where to move nightwatch tests to:

1. We can move accessibility testing to phpunit + axe

2. For actually writing tests in js, looks like cypress and playwright have the same capabilities as nightwatch, but even within Drupal contrib and custom projects are being used instead of nightwatch, let alone the wider js ecosystem. and/or jest for js unit tests. Not sure if there's a way to get actual numbers for that though?

@nod_ that toolbar test is going to be deprecated and removed along with toolbar module before too long, so for me it's more of a question whether we need similar testing for the navigation module, and if so do we want to be writing those in nightwatch or something else.

I think moving the accessibility tests to phpunit + axe is a good idea in its own right, then we're left with tests we genuinely can't do (or not easily) in phpunit and it will be less to convert to cypress or playwright if we end up doing something like that.

works for me

cypress and playwright have the same capabilities as nightwatch, but even within Drupal contrib and custom projects are being used instead of nightwatch, let alone the wider js ecosystem. and/or jest for js unit tests. Not sure if there's a way to get actual numbers for that though?

There might be a way to query d.o.'s GitLab to get the numbers for Drupal contrib, but within the wider JS ecosystem, looks like both Cypress and Playwright are 50x more popular than NIghtwatch, and Playwright's adoption is growing faster than Cypress and surpassed it a few months ago: https://npmtrends.com/cypress-vs-nightwatch-vs-playwright.

We use mocha and playwright for @drupal/once btw

I had initially wanted to try Cypress for Project Browser, but we ended up with FunctionalJavascript tests for most - but they are flaky consistently. Not sure if it's the tests themselves, the framework, or what. We had a plan to convert to Nightwatch just to see if they were LESS flaky, so this impacts us quite a bit, depending on the decision. Being able to use a JS testing framework (whether that be Cypress, Nightwatch, or Playwright) would be fine, though. knowing that Playwright is already in play (🥁) for drupal/@once is enough of a reason to lean that way.

At drunomics we tried using cypress in the past, but had problems with cross-iframe and browser window tests, since cypress runs in brower-context. playwright does not face that limitation, so we are using that now. So far experience with playwright has been great, things like playwright traces that can be automatically be created on fail are a huge time-saver. -> https://playwright.dev/docs/trace-viewer

We've been using Cypress via TugBoat for Mercury Editor → and our Prototype → theme. I haven't personally used Playwright yet. I'm excited to try it out soon as Mercury Editor requires extensive testing across iframes and from what I've read, Playwright handles that better than Cypress out of the box.

Both Experience Builder and Drupal CMS are using Cypress, but I have been unable to find any record of a decision to choose one over the other. I understand that Drupal CMS is using Cypress because XB is (fair enough), but that leaves the XB decision which there is no record of. The furthest back I can see is https://git.drupalcode.org/project/experience_builder/-/merge_requests/5 which has no issue associated and no high level discussion on the MR.

On this issue, it looks like most people have no preference (apart from broad agreement we want to replace nightwatch, no-one wants to keep it), but that all the people that do have a preference prefer playwright, either because they've used both and had less issues with playwright, or because they've used Cypress, and ran into issues that they understand Playwright solves.

If we look at https://npmtrends.com/cypress-vs-playwright, Cypress usage has flattened out in the past year, whereas playwright has increased four-fold and strongly overtaken it. That could change again if something else shows up, but it doesn't look like it will reverse.

So - can we just pick playwright here? The quicker we can make a decision, the less tests to convert for anyone that wants to align with what core uses.

Still think we should get axe + functional js testing working, but that's a separate thing to sort out, even if some individual nightwatch tests might move over to it.

+1 Playwright. Doesn't have a plugin and command system like Cypress, making it easier to work with. The locators it provides out of the box are much nicer and enforce accessibility based on title, labels, etc. Converted a work project from Nightwatch to Playwright and it's been amazing.

Playwright still has a UI and tracing capabilities. It's not backed by a company selling Cloud offerings. Cypress is the new age Saucelabs + Selenium. Selenium was a long living success but not as confident about Cypress.

Per #14, Cypress has really nice time-travel debugging. Turns out, a couple weeks before I wrote #14, Cypress wrote a blog post about that in https://www.cypress.io/blog/debugging-with-cypress-vs-playwright. For people who've used those debugging tools in Cypress and who've also used Playwright, how would you compare your debugging experience between the two? Obviously that blog post is biased, so I'd love to read some unbiased opinions about this. I haven't used Playwright myself yet, so it's not something I can speak to, other than seeing how nice Cypress is for that and not wanting to lose that without Playwright offering a comparable level of debugging productivity.

I understand that Drupal CMS is using Cypress because XB is (fair enough), but that leaves the XB decision which there is no record of.

For XB, we did not evaluate Cypress vs Playwright. We chose Cypress only because several members of the team had experience with Cypress and liked it, and no one had experience with Playwright. Our experience with Cypress is positive, but that doesn't mean we wouldn't also have a positive experience with Playwright.

Note that Cypress Cloud (paid service) recently added a UI coverage feature. We are not currently using Cypress Cloud on XB, so switching to Playwright wouldn't lose us anything with respect to that relative to our status quo. However, if that feature actually does what it says it does, that seems like a potentially awesome tool. Given that it requires a commercial subscription, not sure how much that should factor into our decision for Drupal core.

@effulgentsia Cypress cloud adding that as a premium feature potentially means it's less likely to be available as a FOSS option I would think.

Someone worked on playwright code coverage with Jest here, the post is four years old so there might be better resources, but it's what I found: https://medium.com/@novyludek/code-coverage-of-e2e-tests-with-playwright...

I'm sure there's basic FOSS code coverage tools available for both Playwright and Cypress. What's unique about Cypress's UI Coverage feature is that it connects the coverage information to the UI that you're building, so you can just look at the UI that you're building and see visually the UI elements and states that aren't getting hit by tests.

I agree that the fact that that's a premium feature makes it less compelling for weighing Cypress vs Playwright. However, I don't think it's out of the realm of possibility that the DA could negotiate a reasonable price for making that available for integration with tests run on drupal.org. Or, that organizations building JS-heavy websites with Drupal would find it worth the price of a Cypress Cloud subscription in order to have that feature for testing their sites.

I suspect that none of this outweighs the simple fact that Microsoft is behind Playwright, and once Microsoft enters a market and gets to 51% market share, then it's only a matter of time before it's game over for the small businesses in that market. In that sense, Playwright is probably the safer choice for Drupal, even though I'm personally fond of some of the unique things that Cypress offers. Also, it does seem like most of the people in this issue find some of the technical advantages of Playwright over Cypress to be compelling.

+1 for Playwright. I work with both Cypress and Playwright and, though Cypress is still an excellent tool, Playwright has pulled ahead. Note that I have no experience with Nightwatch so my comments are restricted just to these two tools.

I just delivered this talk at BADCamp 2024:
Should I Choose Cypress or Playwright?
https://www.youtube.com/watch?v=TXTtzxSX9Ns

And, for those who want to do a deep dive, I've collected almost all the comparisons that exist between the two tools here:
Learning Resources
https://performantlabs.com/automated-testing-kit/learning-resources

It's not necessary to watch all of the comparisons (though I have). Watching/reading the first two on the list will do the job—plus my video above.

Here is a brief summary.

In my view, Playwright has caught up in almost all areas to Cypress (Playwright was released later) and has now surpassed it. The debugging plugin for Visual Studio Code makes the test/debug loop very efficient and is a joy to use.

Playwright doesn't have Cypress-isms, which are the equivalent to our Drupalisms. These Cypress-isms exist because of fundamental design choices Cypress made that can never be un-made. (I explain in my video.) Playwright does not have these issues.

Cypress usage is slightly dropping while Playwright usage is now double that of Cypress and is growing at an astonishing rate. I think the reason Cypress usage is dropping off is because teams are switching to Playwright—it really is that much better. Graphs are in the BADCamp video.

Playwright is MUCH faster than Cypress. Timings in the video.

I really loved Cypress when I discovered it. They really modernized E2E testing a huge amount. I started writing all the tests for Automated Testing Kit (https://drupal.org/project/automated_testing_kit) in Cypress and it was only reluctantly that I experimented with Playwright. Once I saw all the advantages of Playwright, I proceeded to re-write all the tests in Playwright so the Kit now has the same tests in both frameworks. I foresee that the Playwright tests will always be ahead of the Cypress tests going forward.

I always recommend that new teams use Playwright now and at least one team was able to re-write their 200 tests with the help of Copilot in just two weeks. Their tests now run about 75% faster than with Cypress (links in the video).

There is one error in the video: it's possible to do parallelization in Cypress reasonably easily with the cypress-split plugin (https://github.com/bahmutov/cypress-split). It has matured a lot and was made by Gleb Bahmutov, who is an expert with Cypress.

I think the BADCamp presentation and Learning Resources page will give those with deep interest everything they may want to know but I am happy to share more insights if they don't.

Another emerging option is vitest in browser mode https://vitest.dev/guide/browser/ - it runs on top of webdriver or playwright

@aangel Thanks for your fantastic comment rooted in real-world experience! 🙏

Playwright is MUCH faster than Cypress.

Fascinating! I'm curious why/how that is. Do you happen to have a URL handy that explains this?

I always recommend that new teams use Playwright now and at least one team was able to re-write their 200 tests with the help of Copilot in just two weeks. Their tests now run about 75% faster than with Cypress (links in the video).

🤯 That'd be the first actually productive use of AI I've personally witnessed in a project! I'd love to see this applied to https://www.drupal.org/project/experience_builder → . Do you happen to also have a URL handy for how that team achieved that?

@wim my pleasure.

The speed difference is due to a few things.

First, the Cypress test runner operates in the browser and typically browsers run JS much slower than, say, node on the OS level. Playwright runs on the OS level then communicates with the browser to issue commands (via CDP or WebKit Protocol).

Be aware that there are some advantages to having the test runner operate within the browser—one has complete access to the code being tested (especially useful for testing front-end JS code). Unfortunately, this complete access comes at the cost of execution time.

Second, Cypress has to re-start the browser for every test.

Third, the method Playwright uses to auto-wait for elements to appear is more efficient than the one Cypress uses.

There might be other reasons but these seem to be the biggest ones.

I'm working on getting a demo of Automated Testing Kit onto DrupalForge.org and SimplyTest. me; hopefully I can post a link
"soon."

BTW, it's not an outlier that tests run 75% faster on Playwright—it's common.

Here is the team that switched:

How We Reduced Testing Time By 70% — by Moving From Cypress to Playwright
Lisa Weilguni, Oct. 2023

https://medium.com/lingvano/how-we-reduced-testing-time-by-70-by-moving-...

Cypress vs Playwright: 70% Faster Automation
The Test Automation Experience, Jan. 2024

https://youtu.be/n6UYHeZBMiY

Their timing improvement:

┌────────────────────────┬──────────┬────────────┬────────────┐
│          Task          │ Cypress  │ Playwright │ Difference │
├────────────────────────┼──────────┼────────────┼────────────┤
│ Run tests locally      │ 1min 27s │ 25s        │ -72%       │
│ Run tests on GitHub CI │ 6min 38s │ 3min 43s   │ -44%       │
└────────────────────────┴──────────┴────────────┴────────────┘

Just been pointed to https://www.drupal.org/project/automated_testing_kit → by @damienmckenna which is trying to build out interoperable common tests for both Cypress and Playwright for common Drupal tasks. Haven't reviewed it yet or anything but making a note.

I should mention that I'm going to convert the current tests in ATK (almost two dozen right now) to work on Drupal CMS.

And at BADCamp, during the Starshot Summit (which was, as you can imagine, well-attended), Tim mentioned that there will be another chance to create Starshot initiatives. I floated the idea of one that creates even more E2E tests than I'll be able to make on my own with my team. The opportunity to create more initiative somes up at the end of November.

But I'm certain I want to do all that in Playwright and will advocate for using it if this does become an initiative. Playwright is very fun to work with. :-)