[shashwat-deep]

Second step of the evidence half (docs/plan.md, "Your half", item 3). Stacked on #2: merge #2 first; GitHub then moves this PR onto main.

What is in it

• eval/score.py scores a detections file (docs/contracts.md §4) against the manifest, as part B of the protocol defines it. Padding and threshold are applied here, so sweeping them needs no new detection run.

  • A mask is a box at or above the threshold, grown by pad, rounded outward to whole pixels and clipped: the pixels the app paints.
  • Coverage is the share of the face's box inside those pixels, computed exactly (the union of masks is cut into cells and the face is clipped to each). A face under the rotation transform is a rotated quadrilateral, which needs no special case.
  • It reports the miss rate (90% rule), the share covered completely, recall at IoU 0.5 (one-to-one, boxes in order of score) and false masks per image (IoU < 0.1 with every face, depictions included). Each comes with a 95% bootstrap interval over images: 10,000 resamples, numpy.random.default_rng(0), percentile interval.
  • Breakdowns: face size (the protocol's five bins), occlusion, truncation, yaw once labelled, and depictions on their own. Each transform and setting in the file is scored separately. --faces writes one row per face for later breakdowns (luma, for example).
  • --operating-point applies the protocol's padding and threshold rules on the tune split.

• eval/transforms.py: for now only the geometry of rot<degrees>, written to match Pillow's rotate(expand=True) exactly, including its transposes at 90°, 180° and 270°. The image transforms and the darkening noise parameters come with the runner, before the first run.

• 35 new tests on synthetic boxes, 44 in total. They cover:

  • the 90% line with rounding (a box 89.5 px wide covers 90 px, so the face is protected; 88.6 px covers 89, so it is missed);
  • overlapping masks counted once;
  • matching by score order;
  • false masks next to depictions;
  • a 45° face that is covered completely while its bounding box has IoU 0.48, so the usual recall misses it;
  • the bootstrap;
  • the operating-point rule, which picks threshold 0.31 with pad 0.10 on a made-up case built so that only that answer fits;
  • rotation against Pillow, pixel by pixel, at 8 angles and 3 image shapes.

  A mutation check confirmed that the tests fail when the order by score, the rounding direction or the depiction rule for false masks is broken.

• Speed: on the real manifest with made-up detections, scoring the 800 report images takes about 1 s, and the operating-point search on the 200 tune images also about 1 s.

For both of us to agree before any run

1. Contract 0.2 (docs/contracts.md §2): masks are rounded outward to whole pixels, left and top edges with Math.floor, right and bottom with Math.ceil. This is what maskFor in app/src/pipeline/mask.ts (Skeleton 3/4: cover each face with a padded opaque block #6) already does, so the contract now records it, and score.py counts the same pixels. (fillRect with fractional edges would blend the edge pixels with the face beneath them.)
2. Protocol, part B (clarifications, no values changed):
   • recall at IoU 0.5 matches one to one in order of score, using boxes before padding, clipped to the image;
   • false masks are counted against depictions too;
   • the bootstrap's generator and interval are stated;
   • padding and threshold depend on each other, so the protocol now says how they are set together: for each threshold from 0.05 to 0.99 in steps of 0.01, take the smallest pad meeting the 99% rule, then the lowest threshold whose false masks meet the 0.25 rule with that pad.

How to check

uv pip sync --python .venv/bin/python eval/requirements.txt
.venv/bin/python -m pytest eval/tests

🤖 Generated with Claude Code