Skip to content

Validate the app against real sites; fix what it found - #48

Merged
woutervanwijk merged 6 commits into
mainfrom
claude/app-validation-test-plan-d89a2f
Nov 13, 2026
Merged

woutervanwijk merged 6 commits into
mainfrom
claude/app-validation-test-plan-d89a2f

Conversation

@woutervanwijk

@woutervanwijk woutervanwijk commented Nov 13, 2026 •

Copy link
Copy Markdown
Owner

What

A harness that checks whether what the app reports about real sites is true, plus the app fixes it turned up.

  • App pass: integration_test/site_validation_test.dart runs the real macOS app on the sites in validation/sites.yaml. It waits until the page has settled and every analysis is done (+20 s for late scripts), then writes the normal export zip.
  • Reference: validation/reference.py measures each site twice without the app's code (curl with the app's User-Agent, openssl, Playwright WebKit, dig, Team Cymru whois). What differs between the two runs counts as volatile, not as an error.
  • Compare: validation/compare.py writes a site × domain matrix with only the deviations. It also has a consistency domain: tracking cookies ↔ Services detections ↔ Sovereignty privacy pillar, consent tool, grade ↔ score.
  • validation/run.sh runs all three. See validation/README.md.

App fixes (each with a test)

Found on Fix
rtvutrecht.nl Certificate host match now reads the SAN (it read only the CN). The real certificate is a test fixture.
kvk.nl, wikipedia A Set-Cookie without SameSite stays "not set" when WebKit reports None. Before, the same cookie scored as weak only when the browser also saw it.
example.com A null MX (RFC 7505) is no longer shown as an empty mail server.
github.com The Sovereignty CMS check shows where the CMS was found, not a raw source excerpt.
rijksoverheid.nl Sovereignty re-runs when detections change, so a tracker found only by a later cookie also reaches the privacy pillar.
allbirds, techcrunch Social widgets count as trackers in Sovereignty, as on the Privacy tab (decided in review). The widgets check is now "Payments & e-commerce", so they don't count twice.
allbirds.com Exported detections list their cookies, because the evidence list is capped at 12.
— The score thresholds (80/50, 75/45) live in one place (lib/services/derive/score_band.dart) for both verdict text and card colour. A test rejects thresholds hard-coded in widgets.

New consistency tests: test/derive/privacy_consistency_test.dart and test/derive/score_band_test.dart.

Results

Last run on 10 sites: 418 ok, 7 volatile, 16 wrong (first run: 78). Of the 16, 10 are bol.com: Akamai served the app's WebView a block page, and the app analyses it as the site. This is not fixed and needs the main document's status from the browser. The other 6 are cookie, storage and resource-count swings on ad and survey sites. Full triage: docs/audits/2026-11-site-validatie.md.

Added after review

  • Block/challenge pages (bol.com, O1): the browser now reports the main document's HTTP status. lib/services/derive/block_page.dart recognises block pages by that status and by Akamai, Cloudflare, DataDome, HUMAN and Imperva markers. The divergence text no longer claims "the browser loaded the page" when the browser was blocked too. Content analyses (metadata, detections, page weight, sovereignty privacy) show a banner or label instead of presenting the block page as the site. Tested offline with a fixture from the real bol.com block page; in the live re-run bol served the real shop, so the blocked path is not proven live.
  • Pages loaded twice (A9): every bare domain (https://www.rijksoverheid.nl) and every redirect (http://kvk.nl) was loaded a second time after it finished, so trackers fired twice and the second document came from cache. Cause: areUrlsEqual treated "" and "/" as different, BrowserView compared URLs literally, and the issued-load URL was cleared once the load finished. Fix: browserAlreadyHas() in navigation_policy.dart. In the real app the reload count went from 2 per site to 0.
  • Still open (O2): on rijksoverheid.nl the first visit from another page gives 1 cookie, and a reload of the same URL gives 5 (as Playwright does). The app reads document.cookie correctly. Timing, visibility, stored data and the cookie wipe are ruled out; details in docs/audits/2026-11-site-validatie.md.

Notes for review

  • integration_test is a new dev dependency. The harness also needs uv and a one-time playwright install webkit. Run output goes to validation/runs/, which is gitignored.
  • flutter test passes (1365). flutter analyze shows only the existing material.dart deprecation infos.
  • User-visible renames in Sovereignty: "Analytics, ads & marketing…" → "Trackers in the site's jurisdiction", "Widgets, embeds & payments…" → "Payments & e-commerce…". New certificate labels: "SAN matches host", "Host not in certificate (CN or SAN)", "CN differs from host (SAN not available)".

🤖 Generated with Claude Code

woutervanwijk and others added 6 commits November 13, 2026 10:03
Add a site-validation harness: an integration test loads ~10 real sites
in the macOS app and exports each inspection package; reference.py
measures the same sites twice with independent tools (curl, openssl,
Playwright WebKit, dig, Team Cymru); compare.py reports only the
differences, plus cross-page consistency checks on the real exports.

App fixes found by the runs, each with a test:
- Certificate host match reads the SAN, not only the CN.
- A Set-Cookie without SameSite stays "not set" when WebKit reports None.
- A null MX (RFC 7505) is no mail server.
- Sovereignty CMS evidence shows where it was found, not raw source.
- Sovereignty re-runs when detections change (cookie-only trackers).
- Social widgets count as trackers in Sovereignty, as on Privacy.
- Exported detections list their cookies (evidence is capped at 12).
- Score thresholds live in one place (score_band.dart) for verdict text
  and card colour.

Triage: docs/audits/2026-11-site-validatie.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When Akamai blocked both the app's own request and the WKWebView, the app
analysed bol.com's "IP address is blocked" page as if it were the site and
the General tab claimed "the browser loaded the page".

- BrowserSnapshot.mainStatus: the main document's HTTP status, from
  PerformanceNavigationTiming.responseStatus (Chromium) or the WebView's
  main-frame onReceivedHttpError (>= 400; WebKit has no responseStatus).
  Unavailable with a reason otherwise; the newest collect wins on merge.
- derive/block_page.dart: classifyBlockPage — status >= 400, or Cloudflare,
  Akamai, DataDome, HUMAN/PerimeterX, Imperva and generic "access blocked"
  markers in a small DOM (scripts that also run on real pages, such as
  Akamai sbsd, are not markers). Checked against 38 real DOMs from the
  validation runs: only bol's block page matches.
- Divergences: "server blocked, browser loaded the page" only when no block
  page was seen; both blocked -> bothBlocked; browser-only -> browserBlocked.
- Visible: banner on the metadata, media, page size, services, privacy and
  sovereignty tabs; sovereignty privacy pillar not verifiable; export marks
  metadata, detections, structured data and page weight unavailable
  (blockedBySite, original kept under onBlockPage), manifest.blockPage,
  report.html note, browser-probe mainStatus.
- Tests: status parsing, HTTP error fallback, classifier (trimmed real
  bol.com DOM fixture), divergences, store merge, export, banner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
areUrlsEqual treated "" and "/" as different paths, BrowserView compared
URLs literally, and the issued-load URL was cleared once the load
finished. A later rebuild then loaded the page a second time (every bare
domain, every redirect), firing trackers twice and serving the second
document from cache. browserAlreadyHas() now decides, with the issued
URL kept until the next navigation.

Found while checking the rijksoverheid.nl cookie difference (still open,
see docs/audits/2026-11-site-validatie.md O2).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflict in cookies_view.dart: main's verdict text ("No score: not enough
data …", renamed to verdict) with this branch's shared thresholds
(cookieScale from score_band.dart).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@woutervanwijk
woutervanwijk merged commit 4230366 into main Nov 13, 2026
1 check failed
@woutervanwijk
woutervanwijk deleted the claude/app-validation-test-plan-d89a2f branch November 13, 2026 09:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant