test(stealth): measure legacy and stealth visibility against third-party fingerprinting and bot-detection oracles

[vringar]

Stacked on #1154. Split out of it so that PR stays focused on the instrument.

Two experiments that measure how visible the legacy and stealth JS instruments are to off-the-shelf, third-party oracles, rather than to probes we wrote ourselves. Both were built to settle a claim the #1154 description used to make with no artifact behind it. Both refuted it.

What was claimed, and what is actually true

The old #1154 body asserted a differential FingerprintJS oracle showing stealth byte-identical to vanilla while legacy differed. There was no such harness anywhere in git history. Measured:

oracle	legacy vs baseline	stealth vs baseline
FingerprintJS 5.2.0 (42 components)	identical — claim refuted	identical
BotD 2.0.0 + fpscanner 1.0.8 (185 signals)	identical — no signal differs	10 signals differ, all navigator.webdriver, in stealth's favour

FingerprintJS reads values, and the legacy wrappers are value-transparent, so it cannot see legacy at all. Neither can the two OSS bot detectors, for a concrete and documented reason: BotD's only native-toString read is Function.prototype.bind and it never compares the string; its detectErrorTrace provokes an error inside BotD's own code; its bot-globals roster has no getInstrumentJS; and fpscanner's one prototype-shape check reads Navigator.prototype, which legacy never touches because it patches the navigator instance.

This is a coverage property of these oracles, not innocence in the instrument. Measured in the same page load, via a separate namespace never counted as an oracle signal, all four instrumentation tells from Krumnow, Jonker & Karsch (arXiv:2205.08890) fire on legacy and none fire on stealth: window.getInstrumentJS present, [native code] gone from four wrapped natives, a getInstrumentJS/instrumentFunction/< frame atop a provoked stack, and prototype flattening taking HTMLCanvasElement.prototype from 10 own properties to 351.

What this does not claim

Stealth does not make OpenWPM undetectable as automation, and a test asserts that so these modules cannot be repurposed as evidence for the opposite. fpscanner's fast bot detection returns true in all four arms, and the paper's Table 4 display-less signature is present everywhere. Those are automation tells, inherent to running under Selenium in a display-less environment — not something the JS instrument causes or the stealth instrument can fix.

Method

Four arms — plain Firefox (bare Selenium, no extension), OpenWPM baseline, legacy with the project's own collection_fingerprinting preset, and stealth. The plain-Firefox arm exists to attribute signals rather than assume: it differs from OpenWPM baseline on zero signals, which is what licenses calling every firing signal an automation tell.

Each arm runs twice first as a validity gate. Unstable signals are excluded and reported with their observed values — two canvas probes were excluded after a control run outside OpenWPM entirely showed Firefox re-seeds canvas noise per profile, with all anti-fingerprinting prefs off. Consequence, recorded rather than hidden: canvas image data is not measurable in this environment, so if legacy leaks into rendered canvas bytes, these experiments could not see it.

Every arm-set run is listed in the reports, including the discarded ones and why — including a probe defect that would have reported a vacuous canvas comparison, since FingerprintJS v5 deliberately returns "skipped" for canvas on Firefox 120+.

Contents

• test/test_fingerprintjs_stamp.py (6 tests), test/test_bot_detection.py (9 tests) — 15 passing, ~5 min
• test/test_pages/vendor/ — FingerprintJS 5.2.0, BotD 2.0.0, fpscanner 1.0.8, all MIT, vendored byte-identical to npm with SHA-256 asserted in-test so the hashes reproduce upstream. Telemetry disabled where the library has any.
• docs/developers/FingerprintJS-Visibility-Experiment.md, docs/developers/Bot-Detector-Visibility-Experiment.md — in-repo, because tests citing a gitignored report is a broken citation.

Review the reports first; the tests pin what the reports measured.

[Copilot]

Pull request overview

Adds two hermetic, reproducible experiments to measure how detectable OpenWPM’s legacy vs stealth JavaScript instrumentation is to off-the-shelf third-party fingerprinting / bot-detection libraries (FingerprintJS, BotD, fpscanner), with pinned vendored fixtures and in-repo measurement reports backing the updated claims in the stacked stealth work.

Changes:

• Vendor and SHA-pin FingerprintJS 5.2.0, BotD 2.0.0, and fpscanner 1.0.8 bundles for hermetic browser tests (with telemetry/monitoring disabled where applicable).
• Add two probe pages plus two pytest harnesses implementing validity gates, activity controls, and per-signal comparisons across baseline/legacy/stealth (and a plain-Firefox control arm for bot detection).
• Add developer-facing experiment reports documenting method, results, and caveats.

Reviewed changes

Copilot reviewed 6 out of 13 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
test/test_pages/vendor/README.md	Documents provenance, pinning, and hermeticity constraints for vendored oracle bundles.
test/test_pages/vendor/fingerprintjs-5.2.0.umd.min.js	Vendored FingerprintJS oracle bundle (pinned artifact).
test/test_pages/vendor/fingerprintjs-5.2.0.LICENSE.txt	FingerprintJS MIT license text.
test/test_pages/vendor/botd-2.0.0.esm.js	Vendored BotD oracle bundle (pinned artifact).
test/test_pages/vendor/botd-2.0.0.LICENSE.txt	BotD MIT license text.
test/test_pages/vendor/fpscanner-1.0.8.es.js	Vendored fpscanner oracle bundle (pinned artifact).
test/test_pages/vendor/fpscanner-1.0.8.LICENSE.txt	fpscanner MIT license text.
test/test_pages/fingerprintjs_stamp.html	Probe page that runs FingerprintJS and publishes per-component comparison keys.
test/test_pages/bot_detection.html	Probe page that runs BotD + fpscanner and publishes per-signal outputs plus paper probes.
test/test_fingerprintjs_stamp.py	3-arm FingerprintJS harness with bundle integrity check, validity gate, and activity control.
test/test_bot_detection.py	4-arm bot-detection harness (incl. plain Selenium control) with bundle integrity check, validity gate, and classifications.
docs/developers/FingerprintJS-Visibility-Experiment.md	Recorded FingerprintJS experiment write-up and results used to justify pinned expectations.
docs/developers/Bot-Detector-Visibility-Experiment.md	Recorded BotD/fpscanner experiment write-up and results used to justify pinned expectations.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[codecov]

Codecov Report

❌ Patch coverage is 4.51306% with 402 lines in your changes missing coverage. Please review.
✅ Project coverage is 49.87%. Comparing base (823e81e) to head (a368115).

Files with missing lines	Patch %	Lines
test/test_bot_detection.py	3.89%	247 Missing ⚠️
test/test_fingerprintjs_stamp.py	5.48%	155 Missing ⚠️

Additional details and impacted files

@@                        Coverage Diff                        @@
##           feat/stealth-js-instrument-v2    #1225      +/-   ##
=================================================================
- Coverage                          53.71%   49.87%   -3.85%     
=================================================================
  Files                                 42       44       +2     
  Lines                               4969     5390     +421     
=================================================================
+ Hits                                2669     2688      +19     
- Misses                              2300     2702     +402     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.