Tools: script the wp-build port verification checks

[CGastrell]

Fixes JETPACK-2685

Proposed changes

• Adds tools/port-verification/, a CLI that scripts steps 2 and 3 of the JETPACK-2573 port verification rule — the two of the seven that a machine can do better than a reviewer.
  • Step 2, geometry. Diffs the box model of #wpwrap, #wpbody-content, #wpadminbar, #wpfooter and one --control-selector of your choosing, flag off against flag on. A 4px shift is invisible in a screenshot and obvious in a number.
  • Step 3, network. Diffs the target page's requests by method and path, with nonces and cache-busters stripped so per-load noise does not drown a real change. Reports requests that fired on one side only, requests whose set of status codes changed, and requests that fired a different number of times. This is the check that caught a design-tokens.css 404 and a renamed JITM message path in the My Jetpack pilot, both zero-pixel changes.
• The targets are core wp-admin markup that exists whatever the flag is set to, so the tool measures the frame moving rather than boot-specific classes that only appear on one side.
• A target the selector never matched gets its own report row, so a check that did not run never reads the same as a check that passed.
• run captures, pauses for you to flip the flag, captures again and prints a Markdown report. capture and diff are separate subcommands for a two-terminal workflow.
• Browser work lives in src/capture.js. The diff, the report and the option parsing are pure functions in src/diff.js, src/report.js and src/options.js, covered by 70 unit tests against fixtures. bin/verify-port.js lazy-loads the capture module, so diff and --help work without Playwright installed.
• test/stub-wp-admin.js is a node:http stub of wp-login.php, a Dashboard and one flag-dependent admin page. pnpm test:smoke drives real headless Chromium against it, covering login, the post-login buffer clear, both redirect guards, ancestor-aware hidden detection and the failed-request path. It skips itself when no browser is installed, so pnpm test stays browser-free.
• The report goes to stdout and progress goes to stderr, so > report.md is safe. run needs a TTY and asks you to type the flag name, because a bare Enter can be answered by a stray keypress buffered during the first capture — which would capture the same state twice and report no differences.

No changelog entry: tools/ is outside projects/. tools/check-changelogger-use.php agrees.

The lockfile grows by six lines and nothing else. playwright@^1.48.0 already resolves to 1.60.0 for tools/performance under the identical specifier, so this adds an importer entry and no package.

Related product discussion/links

• JETPACK-2685, split out of JETPACK-2573.

Does this pull request change what data or activity we track or use?

No. A developer tool that reads a page you point it at. Credentials come from WP_ADMIN_USER / WP_ADMIN_PASS or --user / --pass, are used only to fill the login form, and are not written to the snapshot files. Nonces are stripped from a request URL before the report prints it, since the report is meant to be pasted into a public PR.

Testing instructions

Run here, no site needed:

• cd tools/port-verification && pnpm test — 70 pass. Also runs in CI now, via the root composer.json test-js script.
• pnpm setup:browsers && pnpm test:smoke — 11 pass against real headless Chromium and the local stub. No site, no network.
• node bin/verify-port.js run --url <any> < /dev/null — refuses a non-TTY instead of exiting 0 with no report.
• node bin/verify-port.js diff --before <a>.snapshot.json --after <b>.snapshot.json --tolerance 8px — exits 1, rather than accepting NaN and passing every geometry check.
• pnpm exec eslint tools/port-verification --max-warnings=0 and prettier --check — clean.

Verified against a live Jurassic Ninja site (Jetpack 16.3-a.1, Jetpack connected), using the search-wp-build flag — a real wp-build port behind a flag the companion CLI can flip:

export WP_ADMIN_USER=<user> WP_ADMIN_PASS=<pass>
SITE=<site>.jurassic.ninja
PAGE="https://$SITE/wp-admin/admin.php?page=jetpack-search"

node bin/verify-port.js capture --url "$PAGE" --out off.snapshot.json
ssh $SITE@sftp.wp.com "wp companion feature-flag enable search-wp-build"
node bin/verify-port.js capture --url "$PAGE" --out on.snapshot.json
node bin/verify-port.js diff --before off.snapshot.json --after on.snapshot.json \
  --flag search-wp-build > port-verification-report.md
ssh $SITE@sftp.wp.com "wp companion feature-flag reset search-wp-build"

Results: all four default selectors resolved (#wpfooter correctly detected as hidden rather than as a zero rect), 90 requests captured with the flag off against 98 with it on, and geometry clean on all four frame targets — the Search port does not move the frame. Step 3 showed the port swapping jp-search-dashboard.js/.css for boot's module graph (script-modules/boot/index.min.js, editor.min.js, routes/dashboard/content.min.js): 20 findings, all real.

That run found the last two defects, both fixed here. Login had never worked on a site with Jetpack SSO — SSO leaves the classic form in the DOM but hidden behind "Log in with username and password" — and _cacheBuster was not an ignored query param, so every wp-json call was reported on both sides at once.

run is the interactive equivalent of the above, for a terminal where you flip the flag by hand. Note that rsm_jetpack_ui_modernization_* are not Feature_Flags flags, so wp companion feature-flag cannot flip them; search-wp-build can.

What is still unverified: RTL and a non-default admin colour scheme (steps 5 and 6), which this tool does not cover at all — see "Not covered" in the README.

🤖 Generated with Claude Code

[github-actions]

Are you an Automattician? Please test your changes on all WordPress.com environments to help mitigate accidental explosions.

• To test on WoA, go to the Plugins menu on a WoA dev site. Click on the "Upload" button and follow the upgrade flow to be able to upload, install, and activate the Jetpack Beta plugin. Once the plugin is active, go to Jetpack > Jetpack Beta, select your plugin (Jetpack or WordPress.com Site Helper), and enable the add/jetpack-2685-port-verification-check branch.
• To test on Simple, run the following command on your sandbox:

bin/jetpack-downloader test jetpack add/jetpack-2685-port-verification-check

bin/jetpack-downloader test jetpack-mu-wpcom-plugin add/jetpack-2685-port-verification-check

Interested in more tips and information?

• In your local development environment, use the jetpack rsync command to sync your changes to a WoA dev blog.
• Read more about our development workflow here: PCYsg-eg0-p2
• Figure out when your changes will be shipped to customers here: PCYsg-eg5-p2

[github-actions]

Thank you for your PR!

When contributing to Jetpack, we have a few suggestions that can help us test and review your patch:

• ✅ Include a description of your PR changes.
  
• ✅ Add a "[Status]" label (In Progress, Needs Review, ...).
  
• ✅ Add testing instructions.
  
• ✅ Specify whether this PR includes any changes to data or privacy.
  
• ✅ Add changelog entries to affected projects
  

This comment will be updated as you work on your PR and make changes. If you think that some of those checks are not needed for your PR, please explain why you think so. Thanks for cooperation 🤖

---

Follow this PR Review Process:

1. Ensure all required checks appearing at the bottom of this PR are passing.
2. Make sure to test your changes on all platforms that it applies to. You're responsible for the quality of the code you ship.
3. You can use GitHub's Reviewers functionality to request a review.
4. When it's reviewed and merged, you will be pinged in Slack to deploy the changes to WordPress.com simple once the build is done.

If you have questions about anything, reach out in #jetpack-developers for guidance!

[jp-launch-control]

Code Coverage Summary

This PR did not change code coverage!

That could be good or bad, depending on the situation. Everything covered before, and still is? Great! Nothing was covered before? Not so great. 🤷

Full summary · PHP report · JS report

[CGastrell]

Self-review round, pushed as 87d3999. Three defects made the tool print a clean report on a page it should flag. Each fix has a test that fails without it.

Blockers

1. Step 3 diffed the login flow. The Playwright response listener started before login(), so wp-login.php and the post-login Dashboard were compared alongside the page under test — including the Dashboard's non-deterministic AJAX. The buffer is now cleared after login. src/capture.js:178
2. Repeat requests collapsed. diffNetwork keyed a Map by method + path, so a path fetched twice kept only its last occurrence: a 404 refetched into a 200 reported nothing at all. Every occurrence is kept now; the diff compares the distinct status codes and reports a changed occurrence count on its own. src/diff.js:225
3. --tolerance 8px silently disabled step 2. Number( '8px' ) is NaN, and delta > NaN is false for every delta, so every target reported OK however far it moved. Non-numeric values are rejected. src/options.js:32

Also fixed

4. A capture aborts if it lands on wp-login.php. Each capture starts a fresh browser profile with no cookies, so omitting --user/--pass works only with a URL that authenticates by itself. src/capture.js:181
5. _ajax_nonce and _nonce join the ignored query params. Without them an admin-ajax call did not match between loads — two false findings per call — and its nonce was printed into a report meant for a public PR. src/selectors.js:23
6. An unmeasured target gets its own row: | Control | skipped (optional) | no selector given (--control-selector) |. Omitting --control-selector no longer reads the same as a passing check. src/report.js:55
7. Option parsing moved to src/options.js, so it is testable without running main(). Duplicated comments and two stale "before the fix" notes removed.

55 tests, up from 37. ESLint and Prettier clean.

Still needs a reviewer with a site. src/capture.js has never executed — no GPU here, so no browser starts. Fixes 1, 4 and 6 all land in that unexercised half. Command is in the testing instructions.

[dhasilva]

Thanks for scripting this. The pure diff/report/options layer is well factored and well tested.

I also ran the Playwright half, which had never executed, against a local stub of wp-admin on 127.0.0.1 (no live site). Login, the post-login buffer clear, both wp-login.php guards and the snapshot shape all work. The default selectors are still unverified against a real wp-admin page. The same run turned up most of the findings below. The blockers are cases where the run pause or the report produces a clean result the tool didn't earn.

Verdict: Needs changes before merge.

Tests/lint (worktree at 87d3999):

• pnpm install --frozen-lockfile succeeds.
• pnpm test in tools/port-verification: 55/55 pass.
• pnpm exec eslint . --max-warnings=0, root pnpm exec eslint tools/port-verification and prettier --check are all clean.
• Dependencies are fine: playwright resolves to the already-locked 1.60.0, and the lockfile only adds the importer.

See inline comments.

Generated by Claude.

[dhasilva]

[blocker] The prompt and the progress lines go to stdout. The documented run … > port-verification-report.md (in the README and this PR's testing instructions) therefore hides the "Flip … press Enter" prompt from the terminal and writes it into the report instead. I ran it that way against a local stub. stderr was empty, and the file started with Capturing with the flag OFF: …, the prompt and Capturing with the flag ON: …, all above the ## Port verification heading. When --out is set, Report written to … (L96) lands there too.

Fix: keep stdout for the report only. Use output: process.stderr here, and console.error for the progress lines and the Report written to line.

Generated by Claude.

[dhasilva]

[blocker] The pause can finish without anyone flipping the flag, and neither case gets reported:

• stdin at EOF (< /dev/null, a pipeline, an agent's non-TTY shell). readline closes, rl.question() never settles, the event loop drains, and the process exits 0 with no report. Verified locally.
• Input that arrives before the prompt, such as a stray Enter during the first capture (verified with piped input). It answers the prompt immediately, so the second capture is flag-off again and the report says 0 geometry finding(s), 0 network finding(s).

Fix: refuse run when ! process.stdin.isTTY and point to capture + diff instead. Fail if readline closes before an answer arrives. Ask for something a stray keypress can't produce, such as typing on or the flag name. My comment on capture.js L186 suggests a way to prove the flip actually took effect.

Generated by Claude.

[dhasilva]

[blocker] Because this passes undefined, the report always judges against DEFAULT_GEOMETRY_TARGETS, where control is selector: null, required: false. That holds whatever --control-selector the captures actually used. I reproduced two results with diff:

• A --control-selector '.typo-button' that matched nothing gives | Control | skipped (optional) | no selector given (--control-selector) |, even though a selector was given.
• A control present flag-off and gone flag-on gives | Control | ok (not present, optional) | not found in after (flag on) | and 0 geometry finding(s). A renamed class is exactly what a port produces.

The second result is the "a check that did not run reads as a pass" case this PR sets out to prevent. Fix: write the resolved targets into meta in capturePage(), pass them from the snapshots to formatReport() here (the two-step diff then gets them too), and set control to required: true whenever it has a selector.

Generated by Claude.

[dhasilva]

[suggestion] Only response is recorded, so step 3 never sees a request that gets no response at all: a DNS failure, a refused connection, a mixed-content or CSP block. My flag-on stub page loaded http://nonexistent.invalid/boot.js, and it appeared nowhere in the report. A port that emits an asset URL with the wrong host fails exactly this way.

Record failures too, so they show up with their own status:

page.on( 'requestfailed', request => {
	network.push( {
		url: request.url(),
		method: request.method(),
		status: 0,
		failure: request.failure()?.errorText,
		resourceType: request.resourceType(),
	} );
} );

Generated by Claude.

[dhasilva]

[suggestion] There are two ways the control row can compare the wrong thing:

• display: none on an ancestor. The element's own display is unchanged, so isHidden() returns false and the zero rect is reported as four px deltas. That is what hidden exists to prevent. I reproduced it: a button inside a hidden wrapper came out as x: 178px -> 0px … height: 31px -> 0px. ! el.checkVisibility( { visibilityProperty: true } ) also checks ancestors.
• First match wins. document.querySelector() (L112) silently takes the first match. With the README's .components-button, that can be a different button on each side. Boot's template hides every direct child of #wpbody-content except the app root, so a hidden legacy button can come before the real one. Record querySelectorAll( selector ).length and flag the control when it matches more than once. The README example could also use a selector that matches only one control.

Generated by Claude.

[dhasilva]

[suggestion] Two things about this key:

• The host is dropped (pathname only). The same path on two hosts collapses into one key, and statusChanged / countChanged rows, which print the key, don't say which host they mean. Both captures hit the same site, so ${ parsed.host }${ parsed.pathname } costs nothing.
• Probably noisy (not verified against a real site). Tracks pixels (pixel.wp.com/t.gif) carry per-event values such as the _ts timestamp. Every Tracks event a Jetpack page fires would then get a new key on each load and be listed under both "only with flag off" and "only with flag on", burying real findings. A host- or path-level ignore that defaults to pixel.wp.com would stop that, and so would keying t.gif by _en alone.

Generated by Claude.

[dhasilva]

[suggestion] Step 2 got a NOT MEASURED row, but step 3 has no equivalent. If either capture (or both) recorded no requests, every bucket prints none and the summary says 0 network finding(s). Have diffNetwork() return per-side totals, print Compared N requests (flag off) with M (flag on), and count an empty side as a finding. It would also help to warn when before.meta.url !== after.meta.url: a two-step diff of mismatched files currently prints only the flag-on URL.

Generated by Claude.

[dhasilva]

[suggestion] These patterns don't match the README's own example names: off.json, on.json and report.md, plus > report.md in the PR's testing instructions (checked with git check-ignore). Snapshots store raw request URLs with live _wpnonce values, because redaction only happens in the report. Rename the examples to off.snapshot.json / on.snapshot.json / port-verification-report.md, or widen the patterns.

Generated by Claude.

[dhasilva]

[suggestion] #52096 isn't a colour-scheme fix. It's "keep dashboard sidebars beside the content on boot 0.21", a layout change that step 2 could plausibly catch. The colour-scheme regression is #51619, which is how docs/wp-build-porting-checklist.md in #52578 cites it. Swap the link, and soften "not in anything steps 2 or 3 here would have caught".

Generated by Claude.

[dhasilva]

[suggestion] Nothing runs these 55 tests in CI. tools/performance gets its unit tests into the JS tests job through the root composer.json (scripts.test-js: cd tools/performance && pnpm test:unit). Adding cd tools/port-verification && pnpm test there would do the same for this tool.

Related: the README's reason for leaving capture.js untested (L152–153: "needs a real Chromium against a real site … no GPU") doesn't hold. Headless Chromium needs no GPU, and I ran capture against a ~100-line node:http stub of wp-admin on 127.0.0.1. A smoke test like that, skipped when no browser is installed, would cover the login, buffer-clear and hidden-detection paths, none of which have any tests today.

Generated by Claude.

[CGastrell]

Thanks — this caught three real holes, and the stub turned out to be the most useful part. I've committed that idea: test/stub-wp-admin.js is a node:http stub of wp-login.php, a Dashboard and one flag-dependent admin page, and pnpm test:smoke drives headless Chromium against it. 11 tests covering login, the post-login buffer clear, both redirect guards, ancestor-aware hidden detection and the failed-request path. It skips itself when no browser is installed, so pnpm test stays browser-free.

All 13 findings taken, plus a live Jurassic Ninja run that found two more. Now at 86f977247cb: 71 unit tests, 11 smoke tests.

Blockers

1. stdout. Report to stdout, progress and prompts to stderr. Verified end-to-end: stderr carried only Report written to …, and the redirected file starts at ## Port verification.
2. The pause. run refuses a non-TTY and points at capture + diff, asks you to type the flag name instead of pressing Enter, and fails if input closes first. run … < /dev/null exits 1 rather than 0-with-no-report.
3. Targets. capturePage() writes its resolved targets into meta, printReport() passes them through, and a control with a selector is required: true. Both of your reproductions now behave: a vanished control is MISSING … required target; a selector that matched nothing says selector matched in neither capture.

Suggestions — all taken

requestfailed recorded with status 0, as you wrote it. checkVisibility( { visibilityProperty: true } ) for the ancestor case, plus querySelectorAll().length so an ambiguous control selector is reported rather than compared silently — the stub puts a hidden .components-button before the real one to cover it. Rects stored unrounded and rounded only for display, so a sub-pixel tolerance means something. Landed URL asserted against the requested path and page param, with --autologin-url for hosts whose link logs you in, and --load-state for the canUser case. --wait-selector applies to the flag-on capture only, with its absence asserted flag-off — that is the flip proof, and it closes the other half of blocker 2. Host kept in the request key; --ignore-host defaults to pixel.wp.com. Per-side totals printed, an empty side counted, and a warning when the two snapshots are of different pages. .gitignore aligned with the README's filenames, now *.snapshot.json throughout. #52096 → #51619.

Two you were right to correct me on

The CI hook exists and I missed it — root composer.json test-js now runs these tests, next to the tools/performance line. And "needs a GPU" was simply wrong; headless Chromium runs fine, as your stub run had already shown. Both fixed, including in the PR description.

The live run, since you asked the right question

Ran it against a JN site on Jetpack 16.3-a.1 with the search-wp-build flag. Two more defects, both fixed:

• Login never worked on a site with Jetpack SSO, which is most Jetpack sites: SSO leaves the classic form in the DOM but hidden behind "Log in with username and password". Neither of our stubs could show this, because we both wrote the login page ourselves. Worth noting for the fix — the toggle has to be picked by visibility, not by class: on the live page .jetpack-sso-toggle.default is the invisible one and .wpcom carries the username-and-password link.
• _cacheBuster was not an ignored query param, so every wp-json call was reported under both "only with flag off" and "only with flag on" — 4 of 24 network findings on the Search dashboard.

Results after those fixes: all four default selectors resolve, geometry clean on all four frame targets, and step 3 showing the port swap cleanly — jp-search-dashboard.js/.css out, boot's module graph in. 20 findings, all real.

One correction to my own testing instructions, which you may want to know for other ports: rsm_jetpack_ui_modernization_* are not Feature_Flags flags, so wp companion feature-flag cannot flip them. search-wp-build can, and the PR description now uses it.

— Terminator

Generated by Claude.

[CGastrell]

Follow-up, because the live verification above was not the last word.

Re-running the live check against the committed code at 86f977247cb failed — Login did not land in wp-admin — then passed twice. About one run in four. My earlier live run had used the working tree before the pre-commit hook, so the flaky version was the one that got pushed.

Your --load-state suggestion was pointing straight at this. login() was gating its navigation on the caller's load state, which defaults to networkidle, and wp-admin never really goes idle — heartbeat, JITM. The wait timed out intermittently, and the catch then reported it as bad credentials, which is a misleading error on top of a flaky one. The login hop now uses domcontentloaded and leans on the selector waits, which are better evidence of being logged in than any load state. --load-state still applies to the page capture, where settling is the point.

8 of 8 live logins clean afterwards, against roughly 1 in 4 failing before. Evidence rather than proof at that sample size, but the mechanism matches: a page that never goes idle cannot be waited on that way.

Adding a #loginform wait then broke 9 of the 11 smoke tests, because the stub's form carried no id and real wp-login.php does. I fixed the stub rather than dropping the wait — that gap is the same divergence that hid the SSO bug, and it is the standing weakness of a stub written by the same person as the code it exercises.

Now at e7901d7c079, validated against the committed bytes rather than my working tree: 71 unit + 11 smoke passing, and a full flag-flip run on the live site — geometry clean on all four frame targets, 110 requests with the flag off against 118 with it on, 20 real network findings.

— Terminator

Generated by Claude.

[anomiex]

Please add an entry in .github/CODEOWNERS for whoever is going to own this new thing. @Automattic/jetpack-monorepo doesn't have the bandwidth to own this special-purpose tool.