Consider dropping Nightwatch in favor of Functional Javascript tests

The thing that this gives us that functional javascript testing doesn't is accessibility testing 📌 Automated A11y tests in Nightwatch Fixed .

However I think this is worth discussing e.g.

1. Is it useful over functional javascript testing for anything other than accessibility testing? If not, should we convert the non-accessibility tests over so that we have minimal nightwatch tests to maintain.

2. Should we try to document better what it's useful for.

In that case it could be worth investigating 📌 Automated A11y tests in PHPUnit Needs work , or seeing if we can use axe-core directly, e.g. via @axe-core/cli or @axe-core/webdriverjs. There is also axe integration with cypress and playwright but I think that's just replacing nightwatch with another similar framework.

I hadn't seen 📌 Automated A11y tests in PHPUnit Needs work but yeah that makes a lot of sense - we could then transfer the nightwatch tests directly over to those.

I've been there ( #3413665-21: Enable modules through Nightwatch API when not testing module enabling → ), but at that time alex pott said:

Re removing Nightwatch - it is doing somethings our PHP based JS testing framework cannot do.

usability testing use Axe - see core/tests/Drupal/Nightwatch/Tests/a11yTestDefault.js
JS unit testing - see core/modules/ckeditor5/tests/src/Nightwatch/Tests/drupalHtmlBuilderTest.js

#3413665-27: Enable modules through Nightwatch API when not testing module enabling →

Thanks @Spokje! We've covered axe in previous comments, but the unit testing bits look like something more suited to jest. The example you provided has a nasty workaround in it already.

General +1 for this and moving unit style JS testing to a more suitable framework (such as jest).

I'm assuming that this is what indicates to Nightwatch that it is a unit test:

'@unitTest': true

There is exactly one occurrence of that in core.

Giving up on that MR, since configuring jest and yarn etc is not my strong suit. But this commit shows what the same test would look like as a jest test.

This is getting a bit off-topic though. Let's focus on the more common usage of Nightwatch.

Worth adding that experience builder is using cypress for testing. No idea why it's using that instead of nightwatch but assuming xb eventually moves to core we definitely shouldn't have all three of functional js, cypress, and nightwatch.

📌 [PP-4] CI: Cypress job should be split in unit vs end-to-end tests Postponed

As expected +1 for dropping Nightwatch and doing axe-stuff elsewhere. I'm convinced all non-JS unit testing can be done (better) in Functional JS Tests.

Worth adding that experience builder is using cypress for testing
[...]
we definitely shouldn't have all three of functional js, cypress, and nightwatch.

Three is definitely a crowd here...
Would be nice to know and maybe even helpful for this issue to known why cypress is chosen.

One of the idea of using nightwatch was to make test writing easier for frontend folks, since it's all JS and you don't need to know a somewhat custom PHP testing tool. Happened to me a couple of times that I was happier writing a nightwatch test instead of having to go for a functionalJS one.

I'm sure we can port most if not all nightwatch tests to functional js tests but at what cost. How would something like toolbarTest.js or toolbarApiTest.js look like? Without even talking about the effort it'd take to convert existing tests instead of working on other things.

Worth adding that experience builder is using cypress for testing. No idea why it's using that instead of nightwatch

Cypress has been really nice in terms of being able to execute the test suite through a UI (not just CLI), and then be able to navigate the browser state at the point of test failure and the stack of states that preceded that failure. Not sure if Nightwatch or other JS testing tools have a similar capability.

We also now have running Cypress (from both CLI and UI) working within DDEV thanks to 📌 DDEV support for Cypress tests Fixed .

OK changing the title here, looks like we have a few options for where to move nightwatch tests to:

1. We can move accessibility testing to phpunit + axe

2. For actually writing tests in js, looks like cypress and playwright have the same capabilities as nightwatch, but even within Drupal contrib and custom projects are being used instead of nightwatch, let alone the wider js ecosystem. and/or jest for js unit tests. Not sure if there's a way to get actual numbers for that though?

@nod_ that toolbar test is going to be deprecated and removed along with toolbar module before too long, so for me it's more of a question whether we need similar testing for the navigation module, and if so do we want to be writing those in nightwatch or something else.

I think moving the accessibility tests to phpunit + axe is a good idea in its own right, then we're left with tests we genuinely can't do (or not easily) in phpunit and it will be less to convert to cypress or playwright if we end up doing something like that.

works for me

cypress and playwright have the same capabilities as nightwatch, but even within Drupal contrib and custom projects are being used instead of nightwatch, let alone the wider js ecosystem. and/or jest for js unit tests. Not sure if there's a way to get actual numbers for that though?

There might be a way to query d.o.'s GitLab to get the numbers for Drupal contrib, but within the wider JS ecosystem, looks like both Cypress and Playwright are 50x more popular than NIghtwatch, and Playwright's adoption is growing faster than Cypress and surpassed it a few months ago: https://npmtrends.com/cypress-vs-nightwatch-vs-playwright.

We use mocha and playwright for @drupal/once btw

I had initially wanted to try Cypress for Project Browser, but we ended up with FunctionalJavascript tests for most - but they are flaky consistently. Not sure if it's the tests themselves, the framework, or what. We had a plan to convert to Nightwatch just to see if they were LESS flaky, so this impacts us quite a bit, depending on the decision. Being able to use a JS testing framework (whether that be Cypress, Nightwatch, or Playwright) would be fine, though. knowing that Playwright is already in play (🥁) for drupal/@once is enough of a reason to lean that way.

At drunomics we tried using cypress in the past, but had problems with cross-iframe and browser window tests, since cypress runs in brower-context. playwright does not face that limitation, so we are using that now. So far experience with playwright has been great, things like playwright traces that can be automatically be created on fail are a huge time-saver. -> https://playwright.dev/docs/trace-viewer

We've been using Cypress via TugBoat for Mercury Editor → and our Prototype → theme. I haven't personally used Playwright yet. I'm excited to try it out soon as Mercury Editor requires extensive testing across iframes and from what I've read, Playwright handles that better than Cypress out of the box.

Both Experience Builder and Drupal CMS are using Cypress, but I have been unable to find any record of a decision to choose one over the other. I understand that Drupal CMS is using Cypress because XB is (fair enough), but that leaves the XB decision which there is no record of. The furthest back I can see is https://git.drupalcode.org/project/experience_builder/-/merge_requests/5 which has no issue associated and no high level discussion on the MR.

On this issue, it looks like most people have no preference (apart from broad agreement we want to replace nightwatch, no-one wants to keep it), but that all the people that do have a preference prefer playwright, either because they've used both and had less issues with playwright, or because they've used Cypress, and ran into issues that they understand Playwright solves.

If we look at https://npmtrends.com/cypress-vs-playwright, Cypress usage has flattened out in the past year, whereas playwright has increased four-fold and strongly overtaken it. That could change again if something else shows up, but it doesn't look like it will reverse.

So - can we just pick playwright here? The quicker we can make a decision, the less tests to convert for anyone that wants to align with what core uses.

Still think we should get axe + functional js testing working, but that's a separate thing to sort out, even if some individual nightwatch tests might move over to it.

+1 Playwright. Doesn't have a plugin and command system like Cypress, making it easier to work with. The locators it provides out of the box are much nicer and enforce accessibility based on title, labels, etc. Converted a work project from Nightwatch to Playwright and it's been amazing.

Playwright still has a UI and tracing capabilities. It's not backed by a company selling Cloud offerings. Cypress is the new age Saucelabs + Selenium. Selenium was a long living success but not as confident about Cypress.

Per #14, Cypress has really nice time-travel debugging. Turns out, a couple weeks before I wrote #14, Cypress wrote a blog post about that in https://www.cypress.io/blog/debugging-with-cypress-vs-playwright. For people who've used those debugging tools in Cypress and who've also used Playwright, how would you compare your debugging experience between the two? Obviously that blog post is biased, so I'd love to read some unbiased opinions about this. I haven't used Playwright myself yet, so it's not something I can speak to, other than seeing how nice Cypress is for that and not wanting to lose that without Playwright offering a comparable level of debugging productivity.

I understand that Drupal CMS is using Cypress because XB is (fair enough), but that leaves the XB decision which there is no record of.

For XB, we did not evaluate Cypress vs Playwright. We chose Cypress only because several members of the team had experience with Cypress and liked it, and no one had experience with Playwright. Our experience with Cypress is positive, but that doesn't mean we wouldn't also have a positive experience with Playwright.

Note that Cypress Cloud (paid service) recently added a UI coverage feature. We are not currently using Cypress Cloud on XB, so switching to Playwright wouldn't lose us anything with respect to that relative to our status quo. However, if that feature actually does what it says it does, that seems like a potentially awesome tool. Given that it requires a commercial subscription, not sure how much that should factor into our decision for Drupal core.

@effulgentsia Cypress cloud adding that as a premium feature potentially means it's less likely to be available as a FOSS option I would think.

Someone worked on playwright code coverage with Jest here, the post is four years old so there might be better resources, but it's what I found: https://medium.com/@novyludek/code-coverage-of-e2e-tests-with-playwright...

I'm sure there's basic FOSS code coverage tools available for both Playwright and Cypress. What's unique about Cypress's UI Coverage feature is that it connects the coverage information to the UI that you're building, so you can just look at the UI that you're building and see visually the UI elements and states that aren't getting hit by tests.

I agree that the fact that that's a premium feature makes it less compelling for weighing Cypress vs Playwright. However, I don't think it's out of the realm of possibility that the DA could negotiate a reasonable price for making that available for integration with tests run on drupal.org. Or, that organizations building JS-heavy websites with Drupal would find it worth the price of a Cypress Cloud subscription in order to have that feature for testing their sites.

I suspect that none of this outweighs the simple fact that Microsoft is behind Playwright, and once Microsoft enters a market and gets to 51% market share, then it's only a matter of time before it's game over for the small businesses in that market. In that sense, Playwright is probably the safer choice for Drupal, even though I'm personally fond of some of the unique things that Cypress offers. Also, it does seem like most of the people in this issue find some of the technical advantages of Playwright over Cypress to be compelling.

+1 for Playwright. I work with both Cypress and Playwright and, though Cypress is still an excellent tool, Playwright has pulled ahead. Note that I have no experience with Nightwatch so my comments are restricted just to these two tools.

I just delivered this talk at BADCamp 2024:
Should I Choose Cypress or Playwright?
https://www.youtube.com/watch?v=TXTtzxSX9Ns

And, for those who want to do a deep dive, I've collected almost all the comparisons that exist between the two tools here:
Learning Resources
https://performantlabs.com/automated-testing-kit/learning-resources

It's not necessary to watch all of the comparisons (though I have). Watching/reading the first two on the list will do the job—plus my video above.

Here is a brief summary.

In my view, Playwright has caught up in almost all areas to Cypress (Playwright was released later) and has now surpassed it. The debugging plugin for Visual Studio Code makes the test/debug loop very efficient and is a joy to use.

Playwright doesn't have Cypress-isms, which are the equivalent to our Drupalisms. These Cypress-isms exist because of fundamental design choices Cypress made that can never be un-made. (I explain in my video.) Playwright does not have these issues.

Cypress usage is slightly dropping while Playwright usage is now double that of Cypress and is growing at an astonishing rate. I think the reason Cypress usage is dropping off is because teams are switching to Playwright—it really is that much better. Graphs are in the BADCamp video.

Playwright is MUCH faster than Cypress. Timings in the video.

I really loved Cypress when I discovered it. They really modernized E2E testing a huge amount. I started writing all the tests for Automated Testing Kit (https://drupal.org/project/automated_testing_kit) in Cypress and it was only reluctantly that I experimented with Playwright. Once I saw all the advantages of Playwright, I proceeded to re-write all the tests in Playwright so the Kit now has the same tests in both frameworks. I foresee that the Playwright tests will always be ahead of the Cypress tests going forward.

I always recommend that new teams use Playwright now and at least one team was able to re-write their 200 tests with the help of Copilot in just two weeks. Their tests now run about 75% faster than with Cypress (links in the video).

There is one error in the video: it's possible to do parallelization in Cypress reasonably easily with the cypress-split plugin (https://github.com/bahmutov/cypress-split). It has matured a lot and was made by Gleb Bahmutov, who is an expert with Cypress.

I think the BADCamp presentation and Learning Resources page will give those with deep interest everything they may want to know but I am happy to share more insights if they don't.

Another emerging option is vitest in browser mode https://vitest.dev/guide/browser/ - it runs on top of webdriver or playwright

@aangel Thanks for your fantastic comment rooted in real-world experience! 🙏

Playwright is MUCH faster than Cypress.

Fascinating! I'm curious why/how that is. Do you happen to have a URL handy that explains this?

I always recommend that new teams use Playwright now and at least one team was able to re-write their 200 tests with the help of Copilot in just two weeks. Their tests now run about 75% faster than with Cypress (links in the video).

🤯 That'd be the first actually productive use of AI I've personally witnessed in a project! I'd love to see this applied to https://www.drupal.org/project/experience_builder → . Do you happen to also have a URL handy for how that team achieved that?

@wim my pleasure.

The speed difference is due to a few things.

First, the Cypress test runner operates in the browser and typically browsers run JS much slower than, say, node on the OS level. Playwright runs on the OS level then communicates with the browser to issue commands (via CDP or WebKit Protocol).

Be aware that there are some advantages to having the test runner operate within the browser—one has complete access to the code being tested (especially useful for testing front-end JS code). Unfortunately, this complete access comes at the cost of execution time.

Second, Cypress has to re-start the browser for every test.

Third, the method Playwright uses to auto-wait for elements to appear is more efficient than the one Cypress uses.

There might be other reasons but these seem to be the biggest ones.

I'm working on getting a demo of Automated Testing Kit onto DrupalForge.org and SimplyTest. me; hopefully I can post a link
"soon."

BTW, it's not an outlier that tests run 75% faster on Playwright—it's common.

Here is the team that switched:

How We Reduced Testing Time By 70% — by Moving From Cypress to Playwright
Lisa Weilguni, Oct. 2023

https://medium.com/lingvano/how-we-reduced-testing-time-by-70-by-moving-...

Cypress vs Playwright: 70% Faster Automation
The Test Automation Experience, Jan. 2024

https://youtu.be/n6UYHeZBMiY

Their timing improvement:

┌────────────────────────┬──────────┬────────────┬────────────┐
│          Task          │ Cypress  │ Playwright │ Difference │
├────────────────────────┼──────────┼────────────┼────────────┤
│ Run tests locally      │ 1min 27s │ 25s        │ -72%       │
│ Run tests on GitHub CI │ 6min 38s │ 3min 43s   │ -44%       │
└────────────────────────┴──────────┴────────────┴────────────┘

Just been pointed to https://www.drupal.org/project/automated_testing_kit → by @damienmckenna which is trying to build out interoperable common tests for both Cypress and Playwright for common Drupal tasks. Haven't reviewed it yet or anything but making a note.

I should mention that I'm going to convert the current tests in ATK (almost two dozen right now) to work on Drupal CMS.

And at BADCamp, during the Starshot Summit (which was, as you can imagine, well-attended), Tim mentioned that there will be another chance to create Starshot initiatives. I floated the idea of one that creates even more E2E tests than I'll be able to make on my own with my team. The opportunity to create more initiative somes up at the end of November.

But I'm certain I want to do all that in Playwright and will advocate for using it if this does become an initiative. Playwright is very fun to work with. :-)

We've standardized on Playwright at Lullabot after evaluating several frameworks, including Nightwatch and Cypress, for reasons mostly covered above. But for anyone using Cypress, take a look at https://sorry-cypress.dev which works well and also has the most hilarious project name!

We have a scaffold we use to run Drupal tests in Playwright, in parallel, inside of ddev. Performance is great - on one project, I could run the entire test suite of ~900 tests locally on my M2 Max Macbook in around 6 or 7 minutes. It's using the same cookie-based routing system for setting up databases, and supports both "fresh install" tests like core and "use an existing database" for much faster visual comparison tests where multiple browsers can safely share a Drupal database.

This certainly isn't close to something that could just come into core (it has a dependency on the Task task runner), but perhaps parts could be useful. Feel free to ping me in Slack or here if there's questions!

https://github.com/Lullabot/playwright-drupal

FunctionalJavascript tests for most - but they are flaky consistently. Not sure if it's the tests themselves, the framework, or what

For all of the test frameworks that market "we solve flakes", we've found that 8 out of 10 times flakes are actually performance and sync bugs in the app and not the fault of the text framework. For example, a CI node was slower and a test interacted with a link before JS had attached. That's rare to happen - but is way more likely with slow phones, bad internet, etc. I don't remember if phpunit has support for this, but playwright has both a --repeat-each and --retries flags which are invaluable debugging locally. If a test passes with --repeat-each 100, you can be pretty confident you've solved the flake!

Oh, and re the issue summary:

Nightwatch testing is extremely unreliable and the cause of many random pipeine test failures.

Our experience across multiple projects is that the underlying framework is almost never to blame for "random" failures. I'm not familiar enough with the core Nightwatch tests to know one way or another. If tests are ported as-is I would expect to see similar random failures. But, I think it's confusing to have two frameworks in core plus a lurking one in Cypress with XB, so standardizing on one, even if it's just "all new frontend tests go in X while we iteratively port tests" would be a win for contributors.

Over on the XB side, we're willing in principle to switch from Cypress to Playwright, but it's not on our short term roadmap to put any time into that. Even writing new tests in Playwright would require non-trivial time to do the initial integration of Playwright and creating the base test classes, utilities, patterns, etc.

If the Drupal CMS team hasn't gone down the Cypress road too far yet (as in, hasn't already written a bunch of Cypress tests), I think it makes a lot of sense for them to switch gears to Playwright, and XB will catch up with that transition eventually.

Based on all the discussion in this issue, it seems like Playwright is the clear favorite, so retitling.

The Drupal CMS team has about 5 or 6 Cypress tests.

I don't think we will be switching to Playwright before our 1.0.0 release; there's too much other stuff to do. If we add Cypress tests in the interim, so be it; I'd rather have more test coverage in a framework we're going to replace, than punt on having tests. We can make it a 1.1.0 target to switch to Playwright.

I've been using a lot more Playwright than Cypress lately. One reason was I needed to get up and run it in CI very quickly, which was not simple in the past with Cypress and do some multi device testing. It wound up being a lot easier to configure and reason about, and better on execution and performance. I think it would be a beneficial add.

Playwright also has a UI mode: https://playwright.dev/docs/test-ui-mode

You can also use the browser Recorder (dev console) to playback and export test specs from the interactions. They aren't always 100% perfect but can save a lot of time.

Issue summary needs updating to indicate that playwright has been picked (makes sense to me given the overwhelming comments in its favour here).

Can I check, would moving to Playwirght solve the issue outlined here: 🐛 Impossible to run only Nightwatch tests in a given directory (f.e. for contrib modules) Needs work ?

thumbs up from me

Agreed

Plus one from me but with a caveat - @mstrelan discussed some of the issues with nightwatch with me during the week (he's been doing a deep dive on random fails 💙) and flagged that a lot of the tests are doing module installs and role creation via the browser which is obviously slow. Ideally we don't repeat that same mistake and instead have more setup scripts that do this via the API.

So, after upgrading to Nighwatch 3.x, most of the listed problems were solved, except one 🐛 Impossible to run only Nightwatch tests in a given directory (f.e. for contrib modules) Needs work that looks for me not critical at all, because we can run tests by tags instead, and it is even more convenient than splitting them per directories.

a lot of the tests are doing module installs and role creation via the browser which is obviously slow.
This is true, but with Playwright or any other JS-based test framework, we will have the same issue, so replacing Nightwatch with Playwright doesn't resolve the problem at all!

Instead, the better solution is prepare API for Browser-based tests on the Drupal side in PHP, and this is exactly what I'm doing in the Test Helpers → module. You can look at the usage examples in the Commercetools → module's Nightwatch tests.

So, for example, to log in as admin you have to use only one line:

  browser.thLogin('user1')

that works pretty quick because makes just one GET request to Drupal. The same is for creating a user:

browser.thCreateUser({ mail, name, password, permissions })

and much more helpers (thDrupalFetchURL(), thSetConfig(), thLogout(), etc.

These Nightwatch commands allow to speed up Nighwatch tests significantly! So, would be glad to see them in Drupal Core.

Also, Nightwatch is only W3C compliant, and has many benefits over Playwright, you can see the comparison table:
https://nightwatchjs.org/guide/comparison-with-leading-frameworks.html

And replacing Nightwatch with Playwright will require rewriting all the tests that we already have for Nightwatch.

So, seems it worth to create a Drupal-related comparison table of the benefits and downsides of this replacement. From my point of view, actually I see no benefits.

Thanks for sharing @murz, that's really interesting. The reason for replacing Nightwatch with Playwright is stated as:

Nightwatch testing is extremely unreliable and the cause of many random pipeline test failures.

So just to make sure, this has improved in 3.x, and there are less random failures, and it is more reliable?

It would be really interesting, if someone ran version 2.x of a Nightwatch test suite, as well as the same tests updated to version 3.x against a code base 50 times, and checked performance and test fail stats :)

Interesting stuff @murz. For the record, I don't have a preference whether we use Nightwatch, Playwright, Cypress or PHPUnit, they're all Selenium under the hood afaik. The main issues I have with the existing tests is the amount of setup required to perform a test. When debugging a failing test recently I found the reason the test failed was because the user didn't have permission for an element to exist. Then I discovered that the setup steps involved installing the site via the UI, logging in as admin, creating a role via the UI, assigning permissions by checking boxes via the UI, logging out, logging in again, creating a user with the new role, logging out, logging in again, enabling a module, logging out then finally logging the user in. In the step where the permissions were assigned, the browser dimensions were not big enough to be able to check the boxes to assign the permissions. The test carried on anyway and then eventually failed. Our Functional JavaScript tests allow us to offload most of that setup to direct API calls in PHP. If we can replicate that in any of these front end test tools then I think we're solving most of the problem.

Anecdotally, I don't think it's the case that there are less Nightwatch random test failures, it's still incredibly frequent (not sure if it's possible to get stats on percentage of failures relative to other jobs from gitlab) - especially when you consider the small amount of Nightwatch tests we actually run vs functional/kernel/functional etc.

See the most recently found issues by @mstrelan here https://git.drupalcode.org/project/drupal/-/merge_requests/11623#note_48... (not necessarily random, but a
new failure after a chromedriver update).

It may be the case that some of the random nightwatch failure are not actually due to nightwatch and due to interderminacy in the tested environment too, which we'd still have to fix with playwright, but it's not the only reason to switch - neither Experience Builder nor Drupal CMS decided to write nightwatch tests, instead choosing Cypress but now thinking about switching to playwright.

For using API instead of the brower to do things there is 📌 Enable modules through Nightwatch API when not testing module enabling Needs work open but it's a bit stalled.

The Needs Review Queue Bot → tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide → to find step-by-step guides for working with issues.