Repository navigation
Validate the app against real sites; fix what it found - #48
Merged
Merged
Conversation
Add a site-validation harness: an integration test loads ~10 real sites in the macOS app and exports each inspection package; reference.py measures the same sites twice with independent tools (curl, openssl, Playwright WebKit, dig, Team Cymru); compare.py reports only the differences, plus cross-page consistency checks on the real exports. App fixes found by the runs, each with a test: - Certificate host match reads the SAN, not only the CN. - A Set-Cookie without SameSite stays "not set" when WebKit reports None. - A null MX (RFC 7505) is no mail server. - Sovereignty CMS evidence shows where it was found, not raw source. - Sovereignty re-runs when detections change (cookie-only trackers). - Social widgets count as trackers in Sovereignty, as on Privacy. - Exported detections list their cookies (evidence is capped at 12). - Score thresholds live in one place (score_band.dart) for verdict text and card colour. Triage: docs/audits/2026-11-site-validatie.md Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When Akamai blocked both the app's own request and the WKWebView, the app analysed bol.com's "IP address is blocked" page as if it were the site and the General tab claimed "the browser loaded the page". - BrowserSnapshot.mainStatus: the main document's HTTP status, from PerformanceNavigationTiming.responseStatus (Chromium) or the WebView's main-frame onReceivedHttpError (>= 400; WebKit has no responseStatus). Unavailable with a reason otherwise; the newest collect wins on merge. - derive/block_page.dart: classifyBlockPage — status >= 400, or Cloudflare, Akamai, DataDome, HUMAN/PerimeterX, Imperva and generic "access blocked" markers in a small DOM (scripts that also run on real pages, such as Akamai sbsd, are not markers). Checked against 38 real DOMs from the validation runs: only bol's block page matches. - Divergences: "server blocked, browser loaded the page" only when no block page was seen; both blocked -> bothBlocked; browser-only -> browserBlocked. - Visible: banner on the metadata, media, page size, services, privacy and sovereignty tabs; sovereignty privacy pillar not verifiable; export marks metadata, detections, structured data and page weight unavailable (blockedBySite, original kept under onBlockPage), manifest.blockPage, report.html note, browser-probe mainStatus. - Tests: status parsing, HTTP error fallback, classifier (trimmed real bol.com DOM fixture), divergences, store merge, export, banner. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
areUrlsEqual treated "" and "/" as different paths, BrowserView compared URLs literally, and the issued-load URL was cleared once the load finished. A later rebuild then loaded the page a second time (every bare domain, every redirect), firing trackers twice and serving the second document from cache. browserAlreadyHas() now decides, with the issued URL kept until the next navigation. Found while checking the rijksoverheid.nl cookie difference (still open, see docs/audits/2026-11-site-validatie.md O2). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…alidation-test-plan-d89a2f
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflict in cookies_view.dart: main's verdict text ("No score: not enough
data …", renamed to verdict) with this branch's shared thresholds
(cookieScale from score_band.dart).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A harness that checks whether what the app reports about real sites is true, plus the app fixes it turned up.
integration_test/site_validation_test.dartruns the real macOS app on the sites invalidation/sites.yaml. It waits until the page has settled and every analysis is done (+20 s for late scripts), then writes the normal export zip.validation/reference.pymeasures each site twice without the app's code (curl with the app's User-Agent, openssl, Playwright WebKit, dig, Team Cymru whois). What differs between the two runs counts as volatile, not as an error.validation/compare.pywrites a site × domain matrix with only the deviations. It also has aconsistencydomain: tracking cookies ↔ Services detections ↔ Sovereignty privacy pillar, consent tool, grade ↔ score.validation/run.shruns all three. Seevalidation/README.md.App fixes (each with a test)
None. Before, the same cookie scored as weak only when the browser also saw it.lib/services/derive/score_band.dart) for both verdict text and card colour. A test rejects thresholds hard-coded in widgets.New consistency tests:
test/derive/privacy_consistency_test.dartandtest/derive/score_band_test.dart.Results
Last run on 10 sites: 418 ok, 7 volatile, 16 wrong (first run: 78). Of the 16, 10 are bol.com: Akamai served the app's WebView a block page, and the app analyses it as the site. This is not fixed and needs the main document's status from the browser. The other 6 are cookie, storage and resource-count swings on ad and survey sites. Full triage:
docs/audits/2026-11-site-validatie.md.Added after review
lib/services/derive/block_page.dartrecognises block pages by that status and by Akamai, Cloudflare, DataDome, HUMAN and Imperva markers. The divergence text no longer claims "the browser loaded the page" when the browser was blocked too. Content analyses (metadata, detections, page weight, sovereignty privacy) show a banner or label instead of presenting the block page as the site. Tested offline with a fixture from the real bol.com block page; in the live re-run bol served the real shop, so the blocked path is not proven live.https://www.rijksoverheid.nl) and every redirect (http://kvk.nl) was loaded a second time after it finished, so trackers fired twice and the second document came from cache. Cause:areUrlsEqualtreated""and"/"as different,BrowserViewcompared URLs literally, and the issued-load URL was cleared once the load finished. Fix:browserAlreadyHas()innavigation_policy.dart. In the real app the reload count went from 2 per site to 0.document.cookiecorrectly. Timing, visibility, stored data and the cookie wipe are ruled out; details indocs/audits/2026-11-site-validatie.md.Notes for review
integration_testis a new dev dependency. The harness also needsuvand a one-timeplaywright install webkit. Run output goes tovalidation/runs/, which is gitignored.flutter testpasses (1365).flutter analyzeshows only the existingmaterial.dartdeprecation infos.🤖 Generated with Claude Code