[shashwat-deep]

Next step of the evidence half (docs/plan.md, "Your half", item 4): the tools for the yaw breakdown, which both of us label independently. Stacked on #12, which is stacked on #10, #9 and #2; merge those first.

What is in it

• eval/yaw_sheet.py --labeller <name> draws the sample and makes a labelling page in data/yaw/<name>/:
  • The sample is the protocol's 600 faces: the first 600 report-split faces that are not depictions, in the order of the SHA-256 of solid-mask:yaw:0:<imageId>:<faceId>. Removing an image in the review keeps every other face's place in the order.
  • The page shows one face at a time: a square crop twice the face box's longer side, from the upright image, with yellow ticks at the box's corners and no detector output.
  • Keys 1 to 4 choose frontal, three-quarter, profile or unclear, and Backspace goes back. The page gives the whole labels file to save, or to download.
  • Labelling can stop and go on later, since the page shows only faces the file doesn't label yet. The page warns before it closes with labels not yet saved. The page loads nothing from the network, stores nothing, and never shows the other author's labels.
• eval/yaw_agreement.py compares the two files:
  • It reports Cohen's κ over the four labels, with a 95% interval from the same bootstrap over images that score.py uses (10,000 resamples, seed 0), and κ over the faces neither of us called unclear.
  • It also reports the observed agreement, each author's counts and the confusion table.
  • It writes the disagreements to eval/labels/yaw-disagreements.csv, with one column per author, named after them, and keeps any labels already settled. --write then puts the final labels into the yaw column of faces.csv, and score.py already breaks the miss rate down by that column.
  • Labels for faces that left the sample (image removed in the review) are counted and left out.
• 6 tests, including the κ formula against a worked example (agreement 0.7, chance 0.5, κ = 0.4) and the page driven in headless Chromium (keys, Backspace, the file it gives back). 72 eval tests pass in total.
• On the real manifest, the page for all 600 faces builds in 5 s, with no external address in it. I didn't look at the crops.

For both of us to agree

Part B of the protocol now states the details:

• the sample order, and that labelling starts once the test set is frozen, so the sample can't change under us;
• the crop;
• one file per author in eval/labels/, and neither of us opens the other's until both are done;
• unweighted κ over the four labels, plus κ without "unclear".

🤖 Generated with Claude Code

[06pratyush]

Review notes on #13, merged. These need a follow-up before labelling starts.

1. Crops from unreviewed images. yaw_sheet.py never reads eval/manifest/review.csv, so it cuts face crops from images nobody has reviewed, and a rebuild never deletes old crops. The description says the page for all 600 faces was built on the current draw, which is unreviewed, so those crops are now in data/yaw/ on your machine. Please delete them. Then have yaw_sheet.py refuse until every image in images.csv has keep in review.csv (the frozen set), and empty its output folder before each build.
2. Labels for faces nobody saw. The keydown handler (line 81) accepts auto-repeat events and keys pressed with Ctrl, Alt or Meta, and a clicked label button keeps the focus. So a held key, or a stray Space or Enter after a click, labels the next faces before they appear. Ignore e.repeat and modified keys, and blur the button after a click. Separately, the copy handler (line 93) clears the unsaved flag after any copy, so closing the page after a partial copy loses labels without a warning.
3. κ needs its inputs recorded (CLAUDE.md, rule 5). The summary from yaw_agreement.py records no SHA-256 of the two label files, the disagreements file or the manifest, and no numpy version, and --out is optional (the CLAUDE.md command leaves it out). Please record those as run_info.json does, and give --out a default under results/.

Two things to decide together before labelling:

• The classes are defined by how many eyes are visible, so a face turned to the camera with sunglasses or hair over the eyes counts as "profile". 188 of the 600 sampled faces are marked occluded. Defining the classes by the turn of the head would keep occlusion out of the yaw breakdown.
• The label files go in eval/labels/, which is your folder. So my file would have to be committed into it, and whoever commits first shows their labels in a PR the other has to review while still labelling. Keeping both files in data/yaw/ until both are done, then committing them together (under results/yaw/, say), avoids that.

🤖 Review written with Claude Code