@murz I looked at the comparison chart...it doesn't seem to reflect recent Playwright. For instance, we run mobile web tests all the time. I haven't heard whether Playwright can do mobile apps but is that a requirement for us?
Also, changing testing frameworks is no longer scary with the help of the bots. Earlier in the thread, I posted the experience of a team moving from Cypress to Playwright in just two weeks since AI did most of the translation work.
From personal experience, when people submit tests to us to add to Automated Testing Kit, they submit in either Cypress or Playwright but not both (since they use only one framework in their project) then we use the help of a bot to quickly make the other version and add it to the respective libraries. The translations still need a human to make the final touches and swap in our helper functions but the bulk of the work is done by the bot.
Of course this doesn't have to happen overnight. Nothing wrong with running two test frameworks during the transition period. True, you'll have to look at two reports (or wire up two verifications in the PR workflow) but that's not the end of the world for a few months.
I do agree that modern test frameworks have reduced—but not eliminated—test flakiness. We can perform a burn in test that gives 100% success after 10 tests when we run it locally. However, when we add the public internet, some tests will drop to 70% and then need to be tweaked.
Easily conduct a burn-in on Playwright with the --repeat-each argument. Cypress needs a plug-in to give it that functionality. Don't know about Nightwatch. Remove the grep below and it will conduct a burn-in for every test in the suite.
npx playwright test --grep "@ATK-PW-1201" --repeat-each 10
@mstrelan To help prevent tests running when the db isn't prepared, Automated Testing Kit performs pre-flight tests that query the target site before proceeding. Helps prevent getting a wall of failed tests when, for instance, Reroute Email hasn't been enabled and configured.
The tests are defined in a yml file. The file below has both "stop if untrue" verifications and warnings.
# Preflight tests, which run before all tests, and block further test
# execution in case of failure.
# Each preflight test is a Drush command and a bunch of conditions.
- command: 'pm:list --format=json'
message: 'Automated Testing Kit, QA Accounts must be enabled. (See data/preflightTests.yml for details)'
# Conditions are made against JSON output of the command. Each condition
# can be obligatory (default) or warning (which doesn't block test execution.)
json:
automated_testing_kit.status:
eq: 'Enabled'
qa_accounts.status:
eq: 'Enabled'
login_security.status:
eq: 'Enabled'
warning: |
Login Security is not installed or enabled.
It may affect FedRAMP tests.Agreed...we tend to do everything with tags but it still makes sense to use separate files. We'll move that in the next release.
This was just a test.
@greggles Not sure if I have a preference, either.
There were many details about the process that I had to learn so I undertook documenting the process with Damien's help (the mistakes are mine, of course).
Would you be open to looking it over up to but not including the section on the D7 Security Group?
https://www.drupal.org/project/securitydrupalorg/issues/3541203 📌 Documenting Security/CVE process Active
@damien I added more sections after our last work session.
Cleaning up...thank you for the submission. This was fixed, likely in 2.0.
Fix applied to 2.1.
Thanks for this, @ressa. We will work with you on it and when it's ready convert it to Playwright.
There probably should be a 2.x dev branch to be consistent but I was also trying to work out having a 2.1 branch earlier in the process. It's probably best just to leave things in 2.x-dev and cherry pick whatever is ready into 2.1 when the time comes. I had a vision of something else but maybe it's not better after all.
@ressa the test is failing on the pre-flight checks. Looking through the output, because I'm seeing "drush: command not found," please ensure that Drush is installed locally and on the target server. Drush is used in the tests and for the pre-flight checks.
[Checks if the requirements mention Drush.]
Yes, Drush is properly mentioned as a requirement.
ATK is a the ultimate in white-box testing: it actually communicates with the destination site. That enables the Kit to do fancy things it normally would not be able to. In this XML test, it allows us to run a custom Drush command that confirms the page was regenerated by comparing file timestamps.
If you are just trying to target your local dev box, ensure Drupal is running and the baseUrl is pointing to the actual URL of your installation. We ship with this URL ("http://automated-testing-kit-d10:8888/') but it's almost certainly wrong for your setup.
The Configuration page explains how to get ready to run tests.
Thanks for the report and yes, what you discovered is true...but I removed 10.3 as a supported platform for a few reasons:
1. It's a demo recipe and, as such, people can play with it just as easily on 11.x as 10.x. to get a feel for things
2. I'm looking at using a new recipe feature that won't be available in 10.x; the recipe will require Drupal 11.2+.
So instead of supporting two, diverging platforms, we support just one :-)
@ressa I had always thought that eventually we would have a test for both since both are highly used. I started with XML Sitemap simply because that's what I was using my company website.
Have you modified the XML Sitemap test to use Simple Sitemap? If so, would you be willing to contribute it back to ATK?
aangel → created an issue. See original summary → .
2.0 has been released.
hestenet → credited aangel → .
hestenet → credited aangel → .
jdleonard → credited aangel → .
jdleonard → credited aangel → .
Thanks for the report. I made the change and merged it.
teknorah → credited aangel → .