Automated A11y tests in PHPUnit

MR open for thoughts and reviews
I added a very basic test on Olivero to ensure it works (I never added a dependency so I'm not sure).

This is related to 📌 Automated A11y tests in Nightwatch Fixed
We might discuss if we want both to be integrated or not.

Not sure if this is a valid test but I reused MediaSourceImageTest::testMediaImageSource
Disabled required alt text
Commented out line $page->fillField("{$source_field_id}[0][alt]", ''); so the image had no alt text

Then ran

    $this->drupalGet($this->assertLinkToCreatedMedia());

    $this->executeAxe();;

Expected it to fail but I didn't and just continued along.

Is there something I'm missing?

@smustgrave Thanks for testing! An empty alt text is not a failure. What's forbidden is no alt attribute at all!
It would be a failure if the image was the only content of a link for example.

Now we have https://www.drupal.org/project/drupal/issues/3293469 📌 Automated A11y tests in Nightwatch Fixed

Do we need this issue? Is PHPUnit going to give us different or better results?

Hi @mgifford!

On its current state, I believe that my patch is less interesting that Nightwatch's one because you can only ignore some problems based on their severity. I believe it could be improved by adding a way to alter the axe rules like it's done in JS. I also considered adding some ways to assert expected success or failures in a future version of my patch.

All that to say that I think the only advantage of this one instead of Nightwatch's one would be that some devs could be more comfortable using PHPUnit than Nigthwatch.

Do you think it could be interesting to have both available?

That I don't know. You've got the PHP developers & the JavaScript developers. Do you think PHPUnit is better supported by modules/themes than Nightwatch?

I think there's some split between nightwatch/cypress in the JS testing community. I'll ask around and see if we can get some other thoughts on this.

I'd suggest that we might want to test different things with them.

I think since axe-core package is now available in version 10.1.x (with the nightwatch update) I think a PHP example could be good to have but I am not sure if we need the same setup for an install profile and such.

Is the AccessibilityTest being picked up by the pipeline? I am not seeing it https://dispatcher.drupalci.org/job/drupal_patches/164094/console if it is.

I think it would be great if a11y tests ran automatically for most paths requested by a test. Currently, it appears that Axe tests in Nightwatch are only run for explicitly-defined paths in dedicated tests.

Based on limited experience with Nightwatch, setting up test conditions appears to be easier for PHPUnit functional javascript tests than it is for Nightwatch tests.

It also appears that there are currently more functional javascript tests than Nightwatch tests (at least, going by the number of test files turned up with a quick find): 242 vs 34.

Doing the a11y tests with PHPUnit might help cast the widest net.

In light of #3467492-15: [policy, no patch] Replace Nightwatch with Playwright → , I'm working on this against 11.x and planning to add support for options so that we can use it like we use Nightwatch's axeRun().

I need implementation advice.

The axe core javascript is an npm package.

In a Slack discussion debugging the failed job, @fjgarlin said that core functional javascript tests don't have access to the core/node_modules directory. So, the file must be provided some other way.

It has to be built and the dist files aren't in the repo, so it can't be retrieved directly from the repo.

I didn't find any other cases like this, so I experimented with installing it locally via Composer into vendor by defining a repository in composer.json as follows and then requiring it as a Composer dev dependency. This is working locally.

Is this a viable solution, or are there other preferred solutions?

        {
            "type": "package",
            "package": {
                "name": "npmjs/axe-core",
                "version": "4.10.2",
                "dist": {
                    "url": "https://registry.npmjs.org/axe-core/-/axe-core-4.10.2.tgz",
                    "type": "tar",
                    "shasum": "85228e3e1d8b8532a27659b332e39b7fa0e022df"
                }
            }
        }

Adding to my last comment:

In case it's preferable to commit the axe javascript file into core, AFAICT it is Mozilla Public License 2.0 and is 541K minified.

kentr → changed the visibility of the branch 3338664-automated-a11y-tests to hidden.

Hid the original MR because the new MR builds on it.