Skip to content

Eval: per-face luma and the darkest-tenth breakdown - #14

Merged
06pratyush merged 3 commits into
mainfrom
eval-luma
Oct 28, 2026
Merged

06pratyush merged 3 commits into
mainfrom
eval-luma

Conversation

@shashwat-deep

Copy link
Copy Markdown
Owner

Next step of the evidence half (docs/plan.md, "Your half", item 4): the measured-light breakdown, "the darkest tenth of faces by mean luma inside the box, against the rest" (protocol part B). Stacked on #13, which is stacked on #12, #10, #9 and #2; merge those first.

What is in it

  • eval/face_luma.py measures each face's mean luma and writes eval/manifest/luma.csv, one row per face of faces.csv. Each mean is taken:

  • eval/score.py reads luma.csv when it exists and adds missRateBy.luma: the darkest tenth (the ⌈n/10⌉ faces with the lowest mean luma among the n faces scored, ties broken by image and face ID) against the rest. Both come with bootstrap intervals, plus the cutoff luma. The per-face CSV gains luma and darkestTenth. A luma.csv that doesn't match faces.csv stops the scoring, with a message to rerun face_luma.py.

  • eval/manifest/luma.csv for the current draw (1,797 faces, 12 s to make).

  • 5 tests:

    • the pixel-centre rule;
    • BT.601 on a known patch;
    • a turned image;
    • the breakdown on 20 made-up faces where the two darkest are missed;
    • that the committed luma.csv matches faces.csv.

    77 eval tests pass in total.

On the current draw (before the review)

Among the 1,298 report-split faces that aren't depictions, the darkest tenth is 130 faces with mean luma at or below 56.2. The median face is at 109.5 and the darkest at 8.1. The review will change the draw a little, so rerun face_luma.py after it (the test catches a mismatch).

For both of us to agree

Part B of the protocol now defines luma (BT.601 on the decoded image), "inside the box" (pixel centres), and the darkest tenth (⌈n/10⌉ of the faces scored, ties by ID).

🤖 Generated with Claude Code

shashwat-deep and others added 2 commits October 28, 2026 10:08
face_luma.py measures each face's mean luma (BT.601, as JPEG stores it) over
the pixels whose centres lie inside its box, on the image as Chromium decodes
it, and writes eval/manifest/luma.csv. score.py reads it and breaks the miss
rate down into the darkest tenth of the faces scored and the rest. Part B of
the protocol now gives the definition.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
score.py stops when luma.csv exists but holds no rows, as it does for any
other mismatch with faces.csv, and a test now covers the darkest tenth's
tie-break by image and face ID.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@shashwat-deep
shashwat-deep marked this pull request as ready for review October 28, 2026 04:41
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@06pratyush
06pratyush changed the base branch from eval-yaw to main October 28, 2026 10:27
@06pratyush
06pratyush merged commit 1d29693 into main Oct 28, 2026
2 checks passed
@06pratyush
06pratyush deleted the eval-luma branch October 28, 2026 10:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants