[test] Add Guidepup screen reader tests by aarongarciah · Pull Request #5328 · mui/base-ui

aarongarciah · 2026-07-24T09:18:16Z

Adds a minimal Guidepup screen-reader test using the existing Radio fixture, with local VoiceOver support and Windows/NVDA CI coverage. No VoiceOver CI coverage for now.

CI run: https://github.com/mui/base-ui/actions/runs/30082080229/job/89445735040?pr=5328

pkg-pr-new · 2026-07-24T09:19:56Z

base-ui-tanstack-start

vite-css-base-ui-example

pnpm add https://pkg.pr.new/mui/base-ui/@base-ui/react@5328

pnpm add https://pkg.pr.new/mui/base-ui/@base-ui/utils@5328

commit: 9009534

code-infra-dashboard · 2026-07-24T09:20:52Z

Bundle size

Bundle	Parsed size	Gzip size
@base-ui/react	0B^(0.00%)	0B^(0.00%)

Details of bundle changes

Performance

Total duration: 1,097.34 ms +87.82 ms^(+8.7%) | Renders: 92 ⁽⁺⁰⁾ | Paint: 1,808.21 ms +133.74 ms^(+8.0%)

Test	Duration	Renders
Checkbox mount (500 instances)	78.18 ms 🔺+13.86 ms^(+21.6%)	1 ⁽⁺⁰⁾
Combobox open — 500 items	34.08 ms 🔺+6.29 ms^(+22.6%)	4 ⁽⁺⁰⁾

13 tests within noise — details

Metric alarms

Test	Metric	Change
Checkbox mount (500 instances)	bench:paint	🔺 +23.23 ms

Check out the code infra dashboard for more information about this PR.

netlify · 2026-07-24T09:22:40Z

✅ Deploy Preview for base-ui ready!

Name	Link
🔨 Latest commit	`9009534`
🔍 Latest deploy log	https://app.netlify.com/projects/base-ui/deploys/6a8802ee58dcc6000819690e
😎 Deploy Preview	https://deploy-preview-5328--base-ui.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
🤖 Make changes	Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

aarongarciah · 2026-07-24T10:30:24Z

@@ -0,0 +1,38 @@
+import { expect } from '@playwright/test';
+import { screenReaderTest as test, type ScreenReaderPlaywright } from '@guidepup/playwright';


screenReaderTest automatically uses VoiceOver or NVDA depending on the OS.

aarongarciah · 2026-07-24T11:25:24Z

+
+const MAX_NAVIGATION_STEPS = 10;
+
+async function navigateToItem(


This helper should be extracted as we see fit.

aarongarciah · 2026-07-27T14:55:30Z

  sharp: true
  unrs-resolver: true

+# Screen recording is not needed for the tests and pulls in ffmpeg-static.


More info: https://www.guidepup.dev/docs/guides/automated-environment-setup#recording-setup-macos-only

michaldudak · 2026-08-17T12:03:41Z

PR review

Nothing is merge-blocking, but two test-infrastructure risks and one documentation gap are worth addressing. Installation, formatting, ESLint, project typechecking, and Playwright discovery passed; the historical Windows/NVDA test passed, but VoiceOver was not run because setup modifies macOS accessibility settings. The PR also conflicts with current master, so it needs rebasing and fresh CI.

Bugs (2)

1. 🟠 Refresh the Guidepup versions before relying on macOS support

Location: package.json:72

"@guidepup/guidepup": "0.29.2",
"@guidepup/playwright": "0.18.0",
"@guidepup/setup": "0.24.1",

These versions predate several fixes directly affecting the newly documented macOS workflow:

VoiceOver navigation could miss item-chooser input or move to the top of the window instead of the document (playwright #46, #47).
TCC database writes had no lock retry (setup #63).
Interrupted processes had no best-effort screen-reader teardown (guidepup #138).

Failure scenario: A contributor follows the README, but VoiceOver remains outside the fixture, a transient TCC lock leaves setup incomplete, or an interrupted run leaves VoiceOver and its mounted preferences active.

Fix: Coordinately update to @guidepup/playwright@0.19.1, @guidepup/guidepup@0.33.1, and @guidepup/setup@0.24.6 after the repository’s 72-hour release-age window permits them.

2. 🟠 Allow enough time for Guidepup’s supported retry path

Location: test/screen-reader/playwright.config.ts:11

timeout: 60_000,

The successful Windows job already took 49.5 seconds, leaving little margin. Guidepup’s own example configuration uses a five-minute timeout and one retry because screen-reader startup, application focus, and navigation contain multiple OS-level waits and retries.

Failure scenario: A cold or loaded Windows runner takes slightly longer to start NVDA or focus Chrome, causing Playwright to terminate an otherwise recoverable test at 60 seconds.

Fix: Use the upstream five-minute timeout and consider its single-retry configuration; the surrounding job already has a 15-minute limit.

Docs (1)

1. 🟡 Document or control the required screen-reader language

Location: test/README.md:176

Screen reader tests use Guidepup ... with VoiceOver on macOS or NVDA on Windows.

The test asserts the English word /radio/i, while Guidepup’s VoiceOver navigation also searches for the literal phrase web content. The README does not say that an English screen-reader environment is required.

Failure scenario: VoiceOver or NVDA uses a non-English language and rejects a correctly exposed radio because its localized role announcement does not contain “radio,” or navigation cannot find the localized web-content entry.

Fix: Configure a stable English screen-reader profile, make expectations locale-aware, or explicitly document English as a prerequisite.

Verdict

Approve after nits - no blocking regression, but the stale Guidepup pins and tight timeout should be corrected during the required rebase.

🤖 Review generated with Codex

aarongarciah · 2026-08-17T12:55:33Z

@michaldudak all comments addressed. CI is failing because we have a policy to only allow deps older than 72h.

aarongarciah · 2026-08-21T08:08:27Z

@michaldudak deps now comply with the 72h policy.

michaldudak

LGTM!

aarongarciah added the test label Jul 24, 2026

aarongarciah commented Jul 24, 2026

View reviewed changes

Comment thread pnpm-workspace.yaml Outdated

aarongarciah commented Jul 24, 2026

View reviewed changes

aarongarciah marked this pull request as ready for review July 27, 2026 12:36

aarongarciah requested review from atomiks, flaviendelangle, jjenzz and michaldudak as code owners July 27, 2026 12:36

aarongarciah force-pushed the evaluate-guidepup-for-base-ui branch from 010845f to 598c4ba Compare July 27, 2026 14:50

aarongarciah commented Jul 27, 2026

View reviewed changes

github-actions Bot added the PR: out-of-date The pull request has merge conflicts and can't be merged. label Jul 29, 2026

aarongarciah force-pushed the evaluate-guidepup-for-base-ui branch from 598c4ba to fe0341c Compare August 17, 2026 12:21

github-actions Bot removed the PR: out-of-date The pull request has merge conflicts and can't be merged. label Aug 17, 2026

github-actions Bot added the PR: out-of-date The pull request has merge conflicts and can't be merged. label Aug 17, 2026

aarongarciah force-pushed the evaluate-guidepup-for-base-ui branch from 2cfc47c to be09ab5 Compare August 19, 2026 07:14

github-actions Bot removed the PR: out-of-date The pull request has merge conflicts and can't be merged. label Aug 19, 2026

aarongarciah added 5 commits August 21, 2026 09:40

[test] Add Guidepup screen reader tests

5dfa8c6

[test] Remove expired Guidepup exceptions

4f93da4

[test] Stabilize screen reader tests

eec1ba0

[test] Fix screen reader CI Node version

0cc6d6a

[test] Update Guidepup dependencies

48b357c

aarongarciah force-pushed the evaluate-guidepup-for-base-ui branch from 62c8d7e to 48b357c Compare August 21, 2026 07:42

[test] Dedupe lockfile

9009534

michaldudak approved these changes Aug 24, 2026

View reviewed changes

aarongarciah merged commit 79bbdc0 into mui:master Aug 24, 2026
24 checks passed

Navigation Menu

Uh oh!

Uh oh!

[test] Add Guidepup screen reader tests - #5328

[test] Add Guidepup screen reader tests#5328
aarongarciah merged 6 commits into
mui:masterfrom
aarongarciah:evaluate-guidepup-for-base-ui

aarongarciah commented Jul 24, 2026 •

edited

Loading

Uh oh!

pkg-pr-new Bot commented Jul 24, 2026 •

edited

Loading

Uh oh!

code-infra-dashboard Bot commented Jul 24, 2026 •

edited

Loading

Uh oh!

netlify Bot commented Jul 24, 2026 •

edited

Loading

Uh oh!

Uh oh!

aarongarciah Jul 24, 2026

Uh oh!

aarongarciah Jul 24, 2026

Uh oh!

aarongarciah Jul 27, 2026

Uh oh!

michaldudak commented Aug 17, 2026

Uh oh!

aarongarciah commented Aug 17, 2026

Uh oh!

aarongarciah commented Aug 21, 2026

Uh oh!

michaldudak left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -0,0 +1,38 @@
		import { expect } from '@playwright/test';
		import { screenReaderTest as test, type ScreenReaderPlaywright } from '@guidepup/playwright';


		const MAX_NAVIGATION_STEPS = 10;

		async function navigateToItem(

Uh oh!

Uh oh!

Conversation

aarongarciah commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pkg-pr-new Bot commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

code-infra-dashboard Bot commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bundle size

Performance

Uh oh!

netlify Bot commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for base-ui ready!

Uh oh!

Uh oh!

aarongarciah Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

aarongarciah Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

aarongarciah Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

michaldudak commented Aug 17, 2026

PR review

Bugs (2)

1. 🟠 Refresh the Guidepup versions before relying on macOS support

2. 🟠 Allow enough time for Guidepup’s supported retry path

Docs (1)

1. 🟡 Document or control the required screen-reader language

Verdict

Uh oh!

aarongarciah commented Aug 17, 2026

Uh oh!

aarongarciah commented Aug 21, 2026

Uh oh!

michaldudak left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

aarongarciah commented Jul 24, 2026 •

edited

Loading

pkg-pr-new Bot commented Jul 24, 2026 •

edited

Loading

code-infra-dashboard Bot commented Jul 24, 2026 •

edited

Loading

netlify Bot commented Jul 24, 2026 •

edited

Loading