CI: Playwright ARIA snapshot testing is highly misleading — subsets are considered valid

Created on 29 August 2025, 1 day ago

Overview

Over at 🐛 `noUi: true` SDCs do still show up in the UI Active , I pushed a commit that removed 2 list items from 2 different expected ARIA snapshots.

But tests continued to pass!?! 🤯

Found the root cause:

WTF, the snapshot tests still passed?!

Investigated, and turns out this is because Playwright defaults to saying "yep, this is valid", if the expectation is a subset of the actual! 🤯🫠

Looks like we need /children: deep-equal everywhere? 😅

Out of scope here, opening new issue for it …

This is that issue.

Proposed resolution

Update all Playwright ARIA snapshot matching tests to use strict equality, or preferably, configure this systemwide so that we can't forget to specify /children deep-equal.

(Basically: strictest by default, opt-in to loosening.)

User interface changes

📌 Task
Status

Active

Version

1.0

Component

Code

Created by

🇧🇪Belgium wim leers Ghent 🇧🇪🇪🇺

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Production build 0.71.5 2024