Automated A11y tests in Nightwatch

Based on number #52 to it sounds like a follow-up needs to be filed?

Also could you move the dependency evaluation issue to the issue summary? Thanks!

Also could you move the dependency evaluation issue to the issue summary? Thanks!

I believe the first paragraph of the issue summary already does this? If there's there's something additional that should be there, specify and it can get taken care of.

Which dependency is #56 requesting to evaluate? There are no new dependencies being added in this issue. Automated accessibility testing is a feature of Nightwatch which was added in 2.3.0: https://nightwatchjs.org/guide/using-nightwatch/accessibility-testing.html.

I didn't review any of the config changes in the MR in detail but the approach seems fine. I don't didn't see anything controversial there. Looking forward to having these tests running automatically as part of the test suite! Thanks everyone who worked on this.

#25 contains a dependency evaluation. I guess the issue changed scope since then?

Hiding confusing patches.

I think this should probably be documented both on the accessibility gate and the frontend developer tools?

Also is the release note accurate if the Axe dependency is no longer being added?

NW for docs updates and to clarify the release note: Either this is making better use of something that is already a dependency, or the dependency was a prior approach that was abandoned.

contains a dependency evaluation. I guess the issue changed scope since then?

Yes, the issue changed scope and the issue summary was changed accordingly on Nov 1 2022 → Since that change, the opening sentence of the issue summary has been (and continues to be)

In the time since that discovery, Nightwatch has added Axe support, so this can be done without any new dependencies.

Also is the release note accurate if the Axe dependency is no longer being added?

The release note is accurate. Axe support is now included with Nightwatch.

Either this is making better use of something that is already a dependency, or the dependency was a prior approach that was abandoned.

Neither. When work on this issue started, Nightwatch did not have built-in Axe support. Nightwatch introduced Axe support in version 2.3.6. The overall approach is effectively the same, but can now be done directly via the Nightwatch API.

NW for docs updates and to clarify the CR

The CR is accurate and was written after Nightwatch introduced built in Axe support. While I could add points that clarify what the change isn't, I think it's best and least confusing to report soley on what it is.

Removing needs issue summary update as the tag was added requesting a dependency evaluation and no new dependency is being added.

Removing docs tag as the additions are below. These can get added to the documentation pages when this issue lands.

**Accessiblity Gates**

Will add to https://www.drupal.org/about/core/policies/core-change-policies/core-gat... →

The change does not result in any Nighwatch + Axe test failures.

Drupal's Nightwatch tests include basic accessibliity scans with Axe as of Drupal 10.1. a11yTestAdmin.js scans a set of admin pages using Claro as the theme, and a11yTestDefault.js scans a set of default pages using Olivero as the theme. (These tests can be configured to run with other themes, but Claro/Olivero is what is used in core tests). Like any change to Drupal, all tests must pass for a change to be considered for core inclusion. If a change introdces a failure in an A11y test, it must either be addressed, or there needs to be sufficient evidence that the failure was a false positive, in which case that specific A11y assertion can be bypassed.

More information on the Nightwatch + Axe testing can be found in core/tests/README.md

** Testing Gates**

Will Add to https://www.drupal.org/about/core/policies/core-change-policies/core-gat... →

If a new UI element or pattern is introduced, it should be included somewhere in the Nightwatch accessbility scans.

From Drupal 10.1 forward, Nightwatch runs Axe accessibility scans on a set of admin/default pages. The tests are run using the nightwatch_a11y_testing profile. If a change to Drupal introduces any kind of element/interaction/etc that is distinct enough from existing elements to have different accessiblity concerns, it should be added to one of the pages scanned in the automated accessiblity tests. If it can't be easily added to an existing page scanned in the tests, a new page should be added to the nightwatch_a11y_testing profile

The tests should be added to one (or both) of the two Nightwatch A11y tests: a11yTestAdmin.js, which scans a set of admin pages using Claro as the theme, and/or a11yTestDefault.js, which scans a set of default pages using Olivero as the theme

More information on the Nightwatch + Axe testing can be found in core/tests/README.md

** Frontend Developer Tools**

A line added to https://www.drupal.org/about/core/policies/core-change-policies/frontend... →

Under the yarn command under the "Run Tests" h2 → add "As of Drupal 10.1 Nightwatch tests include basic Axe accessiblity scans "

Very cool. Thanks so much @bnjmnm

In #52 I was just thinking that the accessibility standards documentation → that we proposed as part of the GAAD Pledge (but that haven't been approved by the Coding Standards team).

I made some slight changes to your draft, but I think much of it is going to be basically the same (spellcheck these before posting). I am not sure where we will be able to see the results of these tests on D.o. Also not sure how well this applies to Drupal Modules/Themes. Suspect the Webform module is the first one that might take advantage of this. For the most part, Nightwatch is still largely being used by Core.

Nightwatch-axe tests

From Drupal 10.1 forward, Nightwatch runs Axe accessibility scans on a set of Core admin/default pages. The tests are run using the nightwatch_a11y_testing profile. If a change to Drupal introduces any kind of element/interaction/etc. that is distinct enough from existing elements to have different accessibility concerns, it should be added to one of the pages scanned in the automated accessibility tests. If it can't be easily added to an existing page scanned in the tests, a new page should be added to the nightwatch_a11y_testing profile.

The tests should be added to one (or both) of the two Nightwatch A11y tests: a11yTestAdmin.js, which scans a set of admin pages using Claro as the theme, and/or a11yTestDefault.js, which scans a set of default pages using Olivero as the theme.

It is recommended that where possible, nightwatch-axe tests are added to test dynamic elements that are using Nightwatch. This will expand our accessibility coverage and reduce the need for manual testing.

More information on the Nightwatch + Axe testing can be found in core/tests/README.md

@mgifford confirmed the "needs followup" he added was addressed by the doc snippets I provided above and the tag can be removed. I'm setting it back to needs review as the docs are done and the CR and IS are current and accurate.

Moving this to a task since it's an implementation issue.

Hiding patches to concentrate on MR.

Leaving few thoughts since @bnjmnm reached out for a review.

1. I think we should move the core gate changes to a new issue. That way we can focus on the actual tool that is being added in this issue, not the change control around changing the core gates.
2. I'm not sure about what should we mention in the release notes since this should not have any impact on end users. However, what is there seems accurate.
3. The change records is accurate and seems useful for someone who is also interested in using this tooling to test their theme.
4. It would be helpful to update the docs to include the new a11y specific tags, and the new arguments for selecting a theme.

I'd be comfortable moving this to RTBC after the core gate changes are moved to a separate issue.

@lauriii makes a good point about moving the core gates portion to a separate issue, and I created 📌 [PP-1] Consider updating core gates to include automated Nightwatch A11y tests Postponed as a followup. These tests benefit Drupal accessibility whether or not they're part of a defined gate, and their presence do not alter how existing defined gates work.

Adding 📌 [PP-1] Allow starterkit theme generator tool to clone Olivero Postponed as a related issue as we intend to use the testing methods introduced here.

Seems the MR 2918 has some failures.

Added 🐛 Improper use of aria-label in "System Powered By" block Fixed , which is a newly-caught A11y error by the tests added here. The aria-allowed-attr tests need to be bypassed for the default theme until that issue is resolved.

Seems to be a valid failure. But using this to help test 3349028

All the tests now pass.
Undoing the change 'aria-allowed-attr': { enabled: false }, that @bnjmnm added I can see the error when running

ddev exec -d /var/www/html/web/core yarn test:nightwatch tests/Drupal/Nightwatch/Tests/a11yTestDefault.js

So appears to be doing its job

🐛 Improper use of aria-label in "System Powered By" block Fixed has been committed, we can remove 'aria-allowed-attr': { enabled: false }

Removed the @todos and the rule skipping for the now-closed 🐛 Improper use of aria-label in "System Powered By" block Fixed .

@VladimirAus if an issue needs work could you not merge HEAD without if it isn't accompanied by additional changes working towards completing this? Whoever is working on the work-that-needs-to-be-done can get the MR current with HEAD pretty easily, it doesn't really need to be done as an isolated task.

lauriii → closed merge request !2918

Committed 9e84cb4 and pushed to 10.1.x. Thanks!

EPIC work here! 🤩

Automatically closed - issue fixed for 2 weeks with no activity.