I should mention that I'm going to convert the current tests in ATK (almost two dozen right now) to work on Drupal CMS.
And at BADCamp, during the Starshot Summit (which was, as you can imagine, well-attended), Tim mentioned that there will be another chance to create Starshot initiatives. I floated the idea of one that creates even more E2E tests than I'll be able to make on my own with my team. The opportunity to create more initiative somes up at the end of November.
But I'm certain I want to do all that in Playwright and will advocate for using it if this does become an initiative. Playwright is very fun to work with. :-)
@wim my pleasure.
The speed difference is due to a few things.
First, the Cypress test runner operates in the browser and typically browsers run JS much slower than, say, node on the OS level. Playwright runs on the OS level then communicates with the browser to issue commands (via CDP or WebKit Protocol).
Be aware that there are some advantages to having the test runner operate within the browser—one has complete access to the code being tested (especially useful for testing front-end JS code). Unfortunately, this complete access comes at the cost of execution time.
Second, Cypress has to re-start the browser for every test.
Third, the method Playwright uses to auto-wait for elements to appear is more efficient than the one Cypress uses.
There might be other reasons but these seem to be the biggest ones.
I'm working on getting a demo of Automated Testing Kit onto DrupalForge.org and SimplyTest. me; hopefully I can post a link
"soon."
BTW, it's not an outlier that tests run 75% faster on Playwright—it's common.
Here is the team that switched:
How We Reduced Testing Time By 70% — by Moving From Cypress to Playwright
Lisa Weilguni, Oct. 2023
https://medium.com/lingvano/how-we-reduced-testing-time-by-70-by-moving-...
Cypress vs Playwright: 70% Faster Automation
The Test Automation Experience, Jan. 2024
https://youtu.be/n6UYHeZBMiY
Their timing improvement:
┌────────────────────────┬──────────┬────────────┬────────────┐
│ Task │ Cypress │ Playwright │ Difference │
├────────────────────────┼──────────┼────────────┼────────────┤
│ Run tests locally │ 1min 27s │ 25s │ -72% │
│ Run tests on GitHub CI │ 6min 38s │ 3min 43s │ -44% │
└────────────────────────┴──────────┴────────────┴────────────┘
+1 for Playwright. I work with both Cypress and Playwright and, though Cypress is still an excellent tool, Playwright has pulled ahead. Note that I have no experience with Nightwatch so my comments are restricted just to these two tools.
I just delivered this talk at BADCamp 2024:
Should I Choose Cypress or Playwright?
https://www.youtube.com/watch?v=TXTtzxSX9Ns
And, for those who want to do a deep dive, I've collected almost all the comparisons that exist between the two tools here:
Learning Resources
https://performantlabs.com/automated-testing-kit/learning-resources
It's not necessary to watch all of the comparisons (though I have). Watching/reading the first two on the list will do the job—plus my video above.
Here is a brief summary.
In my view, Playwright has caught up in almost all areas to Cypress (Playwright was released later) and has now surpassed it. The debugging plugin for Visual Studio Code makes the test/debug loop very efficient and is a joy to use.
Playwright doesn't have Cypress-isms, which are the equivalent to our Drupalisms. These Cypress-isms exist because of fundamental design choices Cypress made that can never be un-made. (I explain in my video.) Playwright does not have these issues.
Cypress usage is slightly dropping while Playwright usage is now double that of Cypress and is growing at an astonishing rate. I think the reason Cypress usage is dropping off is because teams are switching to Playwright—it really is that much better. Graphs are in the BADCamp video.
Playwright is MUCH faster than Cypress. Timings in the video.
I really loved Cypress when I discovered it. They really modernized E2E testing a huge amount. I started writing all the tests for Automated Testing Kit (https://drupal.org/project/automated_testing_kit) in Cypress and it was only reluctantly that I experimented with Playwright. Once I saw all the advantages of Playwright, I proceeded to re-write all the tests in Playwright so the Kit now has the same tests in both frameworks. I foresee that the Playwright tests will always be ahead of the Cypress tests going forward.
I always recommend that new teams use Playwright now and at least one team was able to re-write their 200 tests with the help of Copilot in just two weeks. Their tests now run about 75% faster than with Cypress (links in the video).
There is one error in the video: it's possible to do parallelization in Cypress reasonably easily with the cypress-split plugin (https://github.com/bahmutov/cypress-split). It has matured a lot and was made by Gleb Bahmutov, who is an expert with Cypress.
I think the BADCamp presentation and Learning Resources page will give those with deep interest everything they may want to know but I am happy to share more insights if they don't.
Aiming to add to 1.4.
Let's get this into 1.3.
Removed in 1.3. Closing.
mel-miller → credited aangel → .
Agreed. We are about to release 1.3 with the email tests sadly removed (Ethereal has stopped offering their service) and I think I've already removed this but I'll confirm.
mel-miller → credited aangel → .
Alas, Ethereal has discontinued this service. Will look for another one.
Exciting...we use QA Accounts as a dependency of Automated Testing Kit:
https://drupal.org/project/automated_testing_kit
Once this is ready for D11, we can run our tests and release ATK for D11.
mel-miller → credited aangel → .
This made it into 1.2.
This made it into 1.2.
Added to 1.3.
Added to 1.3.
Fixed in 1.3.
Possible to get this merged? Would love to have this working on D11 :-).
I would do both with two different tests.
This looks good to me...you will need a fixture (a data file of dummy data) that you import.
Place the fixture in the data folder and name it feedsData.txt (or whatever suffix you need).
For your tests, use ATK-PW-1180/ATK-CY-1180. You'll have the whole range up to 1189.
Let's back up a moment and recognize that the marketplace has matured since the release of Drupal 7. Frankly, many of the reasons a person might have wanted to use Drupal 7 are now easily handled by Wix, SquareSpace and the others.
At the same time, over time there are always new entrants into any field. Thus, it seems to me that Drupal was always going to lose market share. We are seeing that with electric cars and Tesla. The Chinese EV makers are coming on strong and it's not reasonable to expect Tesla to keep the market share it once had.
Back to the online tools. One often reads accounts online of how people/companies hit a roadblock with these tools that forced them to move to more robust and customizable solutions such as Wordpress or Drupal or the many other options, like Strapi and Contentful.
I see Drupal CMS (the now-official name for Project Starshot) as the perfect solution to handle the many valid concerns raised in the initial post. It's literally taken years for critical components such as automatic updates → and recipes → to come to fruition. I haven't tried automatic updates yet but I've been playing with recipes and they are fantastic.
Add to the above the Experience Builder Initiative → , the modernized themes, the thorough adoption of composer (a package manager is not optional nowadays if one wants access to the wider world of open source libraries) and more and Drupal CMS will be positioned very well.
Agencies will start saying one of two things to smaller entities:
"Dear customer, your needs can easily be handled by the online tools such as Wix and SquareSpace."
or
"Dear customer, you need more headroom than the online tools provide because I foresee some custom programming for this project or in the near future. It doesn't make sense to put you on Wix because we'll simply have to move you off of it in 18 months. We recommend starting with Wordpress/Drupal CMS/Strapi/etc."
In this world, Drupal CMS will be a viable contender. To be clear, it actually isn't right now—I happen to agree with the OP's views.
Can I foretell the future? Not reliably. But I am willing to bet money that by 2027 Drupal CMS will have a thriving ecosystem catering to smaller builds. To further that goal, I for one am going to have a set of end-to-end tests written in Playwright ready for Drupal CMS (aka Starshot) by January 1, 2025.
André Angelantoni
---------------------------
PerformantLabs.com
Check out Automated Testing Kit
https://www.drupal.org/project/automated_testing_kit →
Well, it seems that Wordpress has a big upgrade in its future, too. Their community is currently discussing it. It seems likely to me that they will experience the same thing that Drupal is experiencing.
It seems that there is no way around this. To stay relevant, codebases must be upgraded from time-to-time, it seems to me.
I'll write a more general comment about what Drupal is experiencing further down the comment section.
Using a standard Drupal 10.3 install with no additional modules (not the toolbar, not the new navigation module):
1. Create atk_menu.spec.js. The new test will be "(ATK-PW-1150) Create a new menu item."
2. Visit https://atk.ddev.site:8493/admin/structure/menu/manage/main/add.
3. Add "New top level menu item" + hash pointing to "" and save.
4. On the home page, verify that the new menu item exists.
thejimbirch → credited aangel → .
hestenet → credited aangel → .
hestenet → credited aangel → .
hestenet → credited aangel → .
hestenet → credited aangel → .
hestenet → credited aangel → .
Converts PW-1120.
Updated ATK-PW-1111.
Merged via aa/Convert-ATK-PW-1110.
Working through aa/Convert-ATK-PW-1110.
Merged aa/Convert-modify-fprop.
Closing because this test ID is now reserved for SimpleSiteMap.
Merged via aa/Convert-ATK-PW-1070 but not closing due to outstanding issues.
Merged aa/Convert-ATK-PW-1061.
Closing.
Merged aa/Convert-ATK-PW-1060.
Merged aa/Convert-ATK-PW-1000.
Re-opening.
Merged from aa/Convert-ATK-PW-1030-Reset-password.
Test removed.
Drush 8 does not have delete by email.
Updated .module file and playwright.atk.config.js files.
Added getRandomUser().
Added new functions and updated existing ones atk_commands.js.
aangel → changed the visibility of the branch 3428609-convert-atkcommands.js-functions to hidden.
Most changes in atk_commands.js.
No documentation changes.
Had to put back baseUrl.
Documentation added.
Work for this occurred in atk_commands (logInViaForm) and passed tests after manually adding accounts.
Documentation updated.
Yes, those phpcs warnings were there before and I didn't want to expand the MR with non-issue related changes. I might have fixed a very few things but not many.
I've done some manual testing, added some tests that handle the three types of date fields (I think what I did should work without the Date module being required) and added other filtering tests.
For sure. You can use the 3.0 dev version → now since it uses HED 4.x.
When I tested it everything worked fine but I held off making a release because I was investigating Single Directory Components at the time. I'm not going to switch to SDC just yet so if you use the 3.0 dev version and it works well for you I will cut a full release. Please let me know.
hestenet → credited aangel → .
With the requirement of sending a form_build_id, this doesn't seem to make sense. Getting the form_build_id requires visiting the form anyway, so just submit the form using logInViaForm().
This likely doesn't work with the requirement of passing a form_build_id token. That requires getting the token by visiting the login form first. In that case, simply submit the form.
Done:
- deleteUserWithEmail()
- getUsernameWithEmail()
Have to create the qa_administrator and qa_authenticated accounts since the QA Accounts → module doesn't exist for Drupal 7.
Changed selector and modified deleteUserWithEmail(), getUsernameWithEmail().
Just released 2.0.0-beta1, which installs on PHP 8.1/8.2 and Drupal 10.1/10.2 and uses HED 3.x.
These are all in.
Thanks for the report. Going into the next revision.
aangel → created an issue.