[shashwat-deep]

Third step of the evidence half (docs/plan.md, "Your half", item 2). Stacked on #9, which is stacked on #2: merge those first.

What is in it

• eval/run_detect.py opens the eval page (docs/contracts.md §3) in headless Chromium with Playwright and, for each test image, calls the page's own solidMask.decode() and solidMask.detectFaces() with score threshold 0.05. It writes results/detections/<run-id>.jsonl (§4) and a run_info.json (versions, hashes, timings, git commit, like run_info.json in the MNIST repo).
  • Images are served by the runner on the page's own origin under /__eval__/images/, so connect-src 'self' allows the fetch and nothing leaves the machine. Any request to another origin stops the run.
  • Images that Open Images stores turned (15 in the manifest) get an EXIF orientation tag (Rotation 90 → 8, 180 → 3, 270 → 6), so the page's createImageBitmap(..., {imageOrientation: "from-image"}) turns them upright, as it does a phone photo. Every decoded size is checked against the manifest.
  • Every image is checked against its SHA-256 before the run; a run that stops early leaves only a .part file; run ids are never reused.
  • Two runs with the same versions must give the same boxesSha256 (the SHA-256 of every line after the run line, which names the run). The protocol's reproducibility rule now says this.
• eval/stub/: a stand-in for app/eval.html with the same window.solidMask interface and the app's CSP. Its "detector" returns the bounding box of each solid magenta patch, so the runner and score.py can be tested with known answers. --page stub is the default until the app's eval page exists.
• 4 tests in Chromium (48 in total): a made-up photo stored upright and turned three ways comes back upright with both faces found to within 1 px, and score.py gives a miss rate of 0. A rerun gives identical boxes; changed images are refused; a page that fetches from another site stops the run. Swapping the EXIF codes for 90° and 270° makes the tests fail.
• On the real manifest, the stub run over all 1,000 images took 23 s. Every decoded size matched the manifest.

For Pratyush

• app/eval.html (§3) is what this runner needs from the app: decode, detectFaces, render and info, on a page served by the dev server. No special handling of image URLs is needed, since the runner serves them on the page's origin. Then run_detect.py --page http://localhost:5173/eval.html runs the real detector.
• nmsIoU defaults to 0.3 for YuNet (as in app/src/config.ts) and 0.5 for YOLO5Face.

Not done yet

• Transforms other than none: the rotation and darkening images, and the darkening noise parameters that the protocol fixes in eval/transforms.py before the first run. Those need decisions (fill colour for the enlarged canvas, resampling, noise model) that I'll bring as a separate PR.

How to check

uv pip sync --python .venv/bin/python eval/requirements.txt
.venv/bin/python -m playwright install --only-shell chromium
.venv/bin/python -m pytest eval/tests

🤖 Generated with Claude Code

[06pratyush]

Review note on #10, merged. This needs a follow-up before the first detection run.

An image's own EXIF orientation can flip it against its boxes without the size check noticing. with_orientation() returns the file unchanged when Rotation is 0, and refuses existing EXIF only for the images it turns. The page decodes with imageOrientation: "from-image", so any EXIF Orientation in a file is applied. Orientations 5 to 8 change the decoded size, so the size check catches them. Orientations 2, 3 and 4 (mirrored, or turned 180°) don't, so the face would be upside down or mirrored against its boxes. #12's transforms.upright() doesn't apply EXIF orientation either (Pillow doesn't), so none and rot0 would also start from different pixels for such a file.

No image in the current draw has one: the two camera originals in it, 6ae23606d22bad2d (3780 × 6111) and cce0d9c5636f322e (6000 × 4000), carry Orientation 1 or none. But a redraw after the review could bring in another original. I suggest refusing any image whose EXIF Orientation is present and not 1, in build_manifest.py or in the runner.

Related, from #2: part B and eval/README.md say the image files are "at most 1,024 px on the long side, with all metadata removed". The two originals above are larger and carry EXIF, IPTC, XMP and ICC segments, and #12 found ICC profiles in 430 images. The tiled-setting note ("runs at each image's own size") is also wrong for those two. Please correct the wording too.

🤖 Review written with Claude Code