Eval: yaw labelling page and Cohen's kappa

CLAUDE.md

@@ -78,6 +78,8 @@ uv venv --python 3.13 .venv && uv pip sync --python .venv/bin/python eval/requir
 .venv/bin/python eval/run_detect.py --page URL                   # detections into results/detections/ (--page stub: data/stub-runs/)
 .venv/bin/python eval/run_detect.py --page URL --transforms none,rot45,dark2   # also rotated and darkened copies
 .venv/bin/python eval/check_colour.py                             # the transforms' sRGB images match Chromium's
+.venv/bin/python eval/yaw_sheet.py --labeller NAME             # page for labelling the yaw sample
+.venv/bin/python eval/yaw_agreement.py [--write]               # Cohen's kappa; --write fills faces.csv yaw
 .venv/bin/python eval/score.py RUN.jsonl --operating-point       # pad and threshold, on the tune split
 .venv/bin/python eval/score.py RUN.jsonl --pad P --threshold T   # score the report split
 ```

docs/ai-use.md

@@ -17,3 +17,4 @@ Both authors work with Claude Code (Anthropic). This log records what Claude did
 | 2026-10-28 | Shashwat Deep | Claude Opus 5.5 wrote `eval/run_detect.py`, the stand-in eval page in `eval/stub/` and their tests (4, in headless Chromium); confirmed in Chromium that the EXIF orientation it adds turns images stored turned upright (a mutation check that swaps two orientation codes makes the tests fail); ran the stub over all 1,000 test images to check every decoded size; read Pratyush's `app/` branches without changing them, to match `detect.ts` and `mask.ts`; added the runner's details to part B and F of `docs/eval-protocol.md`. | Not yet. |
 | 2026-10-28 | Shashwat Deep | Claude Opus 5.5 reviewed Pratyush's pull requests #3 to #8: read the app, the tests and the workflow, reran the build, the unit tests and the privacy suite in Chromium, and checked the model's SHA-256 against the OpenCV Zoo and each action's pin against its release. At Shashwat's request it then approved and merged them in order, and resolved the merge conflicts with this pull request in `CLAUDE.md` and this file. | Shashwat read the review and decided to merge. |
 | 2026-10-28 | Shashwat Deep | Claude Opus 5.5 wrote the rotation and darkening transforms in `eval/transforms.py`, `--transforms` in `eval/run_detect.py`, `eval/check_colour.py` and their tests; proposed the parameters written into part B of `docs/eval-protocol.md` (bicubic rotation with mid-grey corners; sensor noise from the middle of the range Brooks et al. fitted, checked against their paper on arXiv); measured that converting with each image's ICC profile matches Chromium's decoded pixels within 1 on 90 test images of 29 kinds of profile, after finding that Chromium applies an RGB profile to a greyscale image. | Not yet. |
+| 2026-10-28 | Shashwat Deep | Claude Opus 5.5 wrote `eval/yaw_sheet.py` (the 600-face yaw sample and a labelling page with face crops and no detector output), `eval/yaw_agreement.py` (Cohen's κ with a bootstrap interval over images, the disagreements to settle, and writing the settled labels into `faces.csv`) and their tests, and added the procedure to part B of `docs/eval-protocol.md`. It checked the κ formula against a worked example and built the page for the real sample without looking at the crops. | Not yet. |

docs/eval-protocol.md

@@ -51,6 +51,10 @@ Each is reported as a miss rate with its 95% CI on the report split.
 - Occlusion and truncation: Open Images' IsOccluded and IsTruncated flags.
 - Depictions (faces on posters, paintings or statues, flagged IsDepiction): reported separately and left out of the headline rate.
 - Yaw, the face's turn away from the camera: a random sample of 600 faces from the report split, each labelled frontal (both eyes visible, face mostly towards the camera), three-quarter (both eyes visible, face clearly turned), profile (one eye or none visible) or unclear. Both authors label independently, from face crops shown without any detector output. We report Cohen's κ, then settle disagreements together. Unclear faces are left out of this breakdown.
+  - The sample is the first 600 report-split faces that are not depictions, in the order of the SHA-256 of `solid-mask:yaw:0:<imageId>:<faceId>`. Labelling starts once the test set is frozen, so the sample no longer changes.
+  - Each crop is a square twice the longer side of the face's box, centred on it, cut from the upright image, with ticks at the corners of the box (`eval/yaw_sheet.py`).
+  - Each author's labels go in a file of their own in `eval/labels/`. Neither author opens the other's file until both are complete.
+  - κ is Cohen's unweighted κ over the four labels, with a 95% confidence interval from the bootstrap over images; we also report κ over the faces neither author called unclear (`eval/yaw_agreement.py`). The settled labels go into the `yaw` column of `eval/manifest/faces.csv`.
 - In-plane rotation: every report-split image rotated counterclockwise by 0°, 15°, 30°, 45°, 90°, 135° and 180° on a canvas enlarged to fit, with the annotated boxes rotated as polygons. The rotation is Pillow's `Image.rotate` with bicubic resampling; the corners it uncovers are mid-grey (128, 128, 128); 90° and 180° are exact transposes. The 0° copy goes through the same steps and is the control for the series.
 - Light, synthetic: every report-split image darkened by 1, 2, 3 and 4 EV: converted to linear RGB (IEC 61966-2-1), scaled by 2^−EV, given Poisson–Gaussian noise, clipped to [0, 1] and converted back to 8-bit sRGB. For a linear value x, the noise is a Poisson count with mean x / λ_shot, times λ_shot, plus Gaussian noise with variance λ_read, so its variance is λ_read + λ_shot·x, the sensor model of Brooks et al. (CVPR 2019, Eq. 1). They fitted log λ_shot to a uniform distribution on [log 0.0001, log 0.012] and log λ_read = 2.18 log λ_shot + 1.2 on the Darmstadt Noise Dataset; we take the middle of that range, λ_shot = 0.0011, and the fitted line, λ_read = 1.17 × 10⁻⁶. The noise is drawn per pixel and channel, with a random generator for each image and level seeded from its ID and the level.
 - Both transforms start from the image as Chromium decodes it: the runner converts it to sRGB with its ICC profile, as Chromium does (`eval/check_colour.py` checks that the pixels agree within 1 for test images with each kind of profile), turns it upright, transforms it and gives it to the page as a PNG.

eval/README.md

@@ -14,6 +14,9 @@ The scripts here measure how often the tool leaves a face uncovered, following [
 - `stub/`: a stand-in for the app's eval page with the same interface, used until that page exists and by the tests.
 - `score.py`: scores a detections file against the manifest: missed faces, faces covered completely, detection recall and false masks, each with a 95% confidence interval, and the operating point.
 - `transforms.py`: the rotation and darkening transforms, their parameters, and the geometry of a rotation, which `score.py` uses.
+- `yaw_sheet.py`: makes a page for labelling the yaw of the sample faces.
+- `yaw_agreement.py`: compares the two authors' yaw labels (Cohen's κ) and writes the settled labels into `manifest/faces.csv`.
+- `labels/`: each author's yaw labels, and the faces they disagreed on.
 - `check_colour.py`: checks that the images the transforms start from have the pixels Chromium decodes.
 - `tests/`: tests for these scripts (pytest).
 
@@ -59,6 +62,15 @@ The runner gives each image to the page itself, on the page's own address, so no
 
 The first finds the padding and threshold on the tune split by the protocol's rule; the second scores the report split at those values. Each transform and setting in the file is scored on its own. A mask covers whole pixels, rounded outward as the app paints it, and a face's covered share is computed exactly from those pixels, so a face turned by a rotation transform needs no special case. The summary is JSON (`--out`); `--faces` also writes one row per face for further breakdowns.
 
+## Yaw labels
+
+Each of us labels the same 600 faces alone, once the test set is frozen ([`docs/eval-protocol.md`](../docs/eval-protocol.md), part B, "Breakdowns"):
+
+1. `.venv/bin/python eval/yaw_sheet.py --labeller <your name>` writes `data/yaw/<your name>/index.html`. Open it, label each face with the keys 1 to 4, and save the box at the bottom as `labels/yaw-<your name>.csv`. You can stop and go on later: the page shows only the faces the file doesn't label yet. Don't open the other author's file until you have both finished.
+2. `.venv/bin/python eval/yaw_agreement.py --out results/yaw-agreement.json` reports Cohen's κ with its confidence interval and writes the faces you disagree on to `labels/yaw-disagreements.csv`.
+3. Settle each of those together and write the agreed label in its `settled` column.
+4. `.venv/bin/python eval/yaw_agreement.py --write` puts the final labels into the `yaw` column of `manifest/faces.csv`, which `score.py` breaks the miss rate down by.
+
 ## How the test set is drawn
 
 `build_manifest.py` downloads Open Images' image lists and box annotations for its validation and test splits and checks them against fixed SHA-256s. It keeps the images that have at least one "Human face" box, no face box marked IsGroupOf, the CC BY 2.0 licence and a known orientation. Each image's split and its place in the draw come from the SHA-256 of its ID, so they don't depend on which other images are eligible. The set is the first 200 tune and the first 800 report images that the review has not removed.

eval/tests/test_yaw.py

@@ -0,0 +1,158 @@
+"""Tests of yaw_sheet.py and yaw_agreement.py on made-up faces and labels."""
+import json
+import numpy as np
+import pytest
+from PIL import Image
+import build_manifest as bm
+import yaw_agreement
+import yaw_sheet
+
+LABELLERS = ['ann', 'bob']
+
+
+def make_manifest(folder, n_images=30, faces_per_image=3):
+    """Made-up images (grey, with a white square per face) and faces; every third face is a depiction."""
+    (folder / 'manifest').mkdir()
+    (folder / 'images').mkdir()
+    images, faces = [], []
+    for k in range(n_images):
+        i = f'{k:016x}'
+        images.append({'imageId': i, 'split': bm.split_of(i), 'url': f'https://example.org/{i}.jpg', 'width': 120,
+                       'height': 90, 'rotation': 0})
+        image = Image.new('RGB', (120, 90), (90, 90, 90))
+        for n in range(faces_per_image):
+            image.paste((255, 255, 255), (10 + 35 * n, 30, 30 + 35 * n, 55))
+            faces.append({'imageId': i, 'faceId': n, 'x': 10 + 35 * n, 'y': 30, 'w': 20, 'h': 25,
+                          'isOccluded': 0, 'isTruncated': 0, 'isDepiction': int(n == 2), 'yaw': ''})
+        image.save(folder / 'images' / f'{i}.jpg', quality=95)
+    bm.write_csv(folder / 'manifest/images.csv', bm.IMAGE_COLUMNS, images)
+    bm.write_csv(folder / 'manifest/faces.csv', bm.FACE_COLUMNS, faces)
+    return ['--manifest', str(folder / 'manifest'), '--labels', str(folder / 'labels')]
+
+
+def manifest_rows(folder):
+    return ({r['imageId']: r for r in bm.read_csv(folder / 'manifest/images.csv')},
+            bm.read_csv(folder / 'manifest/faces.csv'))
+
+
+def write_labels(folder, name, labels):
+    (folder / 'labels').mkdir(exist_ok=True)
+    bm.write_csv(folder / 'labels' / f'yaw-{name}.csv', yaw_sheet.LABEL_COLUMNS,
+                 [{'imageId': i, 'faceId': n, 'yaw': label} for (i, n), label in sorted(labels.items())])
+
+
+def test_sample_takes_report_faces_that_are_not_depictions_in_a_fixed_order(tmp_path):
+    make_manifest(tmp_path)
+    images, faces = manifest_rows(tmp_path)
+    chosen = yaw_sheet.sample(images, faces, n=10)
+    assert len(chosen) == 10 and chosen == yaw_sheet.sample(images, faces, n=10)
+    assert all(images[f['imageId']]['split'] == 'report' and f['isDepiction'] == '0' for f in chosen)
+    gone = chosen[0]['imageId']  # removing an image keeps every other face's place in the order
+    rest = yaw_sheet.sample({i: r for i, r in images.items() if i != gone}, [f for f in faces if f['imageId'] != gone], n=10)
+    kept = [f for f in chosen if f['imageId'] != gone]
+    assert rest[:len(kept)] == kept
+
+
+def test_crop_centres_the_face_and_pads_beyond_the_image():
+    image = Image.new('RGB', (100, 80), (0, 0, 0))
+    image.paste((255, 255, 255), (0, 0, 20, 20))          # a face in the top-left corner
+    out = yaw_sheet.crop(image, {'x': 0, 'y': 0, 'w': 20, 'h': 20})
+    pixels = np.asarray(out).astype(int)
+    assert out.size == (yaw_sheet.CROP, yaw_sheet.CROP)
+    assert (pixels[5, 5] == yaw_sheet.transforms.FILL).all()                  # beyond the image: mid-grey
+    assert (pixels[128, 128] > 240).all()                                     # the face is in the middle
+    assert (np.abs(pixels[61, 61:70] - yaw_sheet.TICK).max(axis=1) < 40).any()  # a tick at the top-left corner
+
+
+def test_sheet_shows_only_faces_not_yet_labelled(tmp_path):
+    args = make_manifest(tmp_path) + ['--images', str(tmp_path / 'images'), '--out', str(tmp_path / 'yaw')]
+    images, faces = manifest_rows(tmp_path)
+    chosen = yaw_sheet.sample(images, faces)
+    done = {(f['imageId'], f['faceId']): 'frontal' for f in chosen[:5]}
+    write_labels(tmp_path, 'ann', done)
+    yaw_sheet.main(args + ['--labeller', 'ann'])
+    page = (tmp_path / 'yaw/ann/index.html').read_text(encoding='utf-8')
+    shown = json.loads(page.split('const faces = ', 1)[1].split(';\n', 1)[0])
+    assert len(shown) == len(chosen) - 5 and not {(f['imageId'], f['faceId']) for f in shown} & set(done)
+    assert all((tmp_path / 'yaw/ann' / f['src']).exists() for f in shown)
+    assert 'bob' not in page
+    with pytest.raises(SystemExit):
+        yaw_sheet.main(args + ['--labeller', 'Ann Smith'])
+    write_labels(tmp_path, 'ann', {**done, ('x', '0'): 'sideways'})
+    with pytest.raises(SystemExit):
+        yaw_sheet.main(args + ['--labeller', 'ann'])
+
+
+def test_kappa_matches_a_worked_example():
+    # 50 faces: both frontal 20, first frontal and second profile 5, the reverse 10, both profile 15. Agreement 0.7,
+    # chance agreement 0.5, so kappa = 0.4.
+    table = np.zeros((4, 4))
+    table[0, 0], table[0, 2], table[2, 0], table[2, 2] = 20, 5, 10, 15
+    assert yaw_agreement.kappa(table) == pytest.approx(0.4)
+    assert yaw_agreement.kappa(np.diag([3, 0, 2, 0])) == 1
+    assert yaw_agreement.kappa(np.zeros((4, 4))) is None and yaw_agreement.kappa(np.diag([5, 0, 0, 0])) is None
+
+
+def test_agreement_reports_kappa_and_keeps_settled_labels(tmp_path, capsys):
+    args = make_manifest(tmp_path) + ['--labellers', *LABELLERS]
+    images, faces = manifest_rows(tmp_path)
+    chosen = [(f['imageId'], f['faceId']) for f in yaw_sheet.sample(images, faces)]
+    rng = np.random.default_rng(0)
+    ann = {face: yaw_sheet.LABELS[k] for face, k in zip(chosen, rng.integers(0, 3, len(chosen)))}
+    bob = dict(ann)
+    for face in chosen[:6]:
+        bob[face] = 'unclear'
+    write_labels(tmp_path, 'ann', ann)
+    write_labels(tmp_path, 'bob', {face: label for face, label in bob.items() if face != chosen[-1]})
+    with pytest.raises(SystemExit, match='1 faces of the sample unlabelled'):
+        yaw_agreement.main(args)
+    write_labels(tmp_path, 'bob', {**bob, ('removed-in-review', '0'): 'frontal'})
+
+    yaw_agreement.main(args + ['--out', str(tmp_path / 'agreement.json')])
+    summary = json.loads((tmp_path / 'agreement.json').read_text())
+    assert summary['labelsOutsideSample'] == {'ann': 0, 'bob': 1}
+    assert summary['faces'] == len(chosen) and summary['disagreements'] == 6 == summary['unsettled']
+    assert summary['observedAgreement'] == pytest.approx(1 - 6 / len(chosen))
+    assert 0 < summary['kappa']['low'] <= summary['kappa']['value'] <= summary['kappa']['high'] <= 1
+    assert summary['kappaWithoutUnclear']['value'] == 1 and summary['kappaWithoutUnclear']['faces'] == len(chosen) - 6
+    assert summary['counts']['bob']['unclear'] == 6 + list(ann.values()).count('unclear')
+    with pytest.raises(SystemExit, match='not settled'):
+        yaw_agreement.main(args + ['--write'])
+
+    disagreements = bm.read_csv(tmp_path / 'labels/yaw-disagreements.csv')
+    assert list(disagreements[0]) == ['imageId', 'faceId', 'ann', 'bob', 'settled']
+    for row in disagreements:
+        row['settled'] = row['ann']
+    bm.write_csv(tmp_path / 'labels/yaw-disagreements.csv', list(disagreements[0]), disagreements)
+    yaw_agreement.main(args[:-3] + ['--labellers', 'bob', 'ann'])  # a rerun, in either order, keeps what was settled
+    assert all(r['settled'] for r in bm.read_csv(tmp_path / 'labels/yaw-disagreements.csv'))
+    yaw_agreement.main(args + ['--write'])
+    written = {(f['imageId'], f['faceId']): f['yaw'] for f in bm.read_csv(tmp_path / 'manifest/faces.csv')}
+    assert {face: written[face] for face in chosen} == ann
+    assert sum(1 for label in written.values() if label) == len(chosen)
+    capsys.readouterr()
+
+
+def test_sheet_page_records_keys_and_backspace(chromium, tmp_path):
+    from playwright.sync_api import sync_playwright
+    args = make_manifest(tmp_path) + ['--images', str(tmp_path / 'images'), '--out', str(tmp_path / 'yaw')]
+    images, faces = manifest_rows(tmp_path)
+    chosen = yaw_sheet.sample(images, faces)
+    write_labels(tmp_path, 'ann', {(chosen[0]['imageId'], chosen[0]['faceId']): 'profile'})
+    yaw_sheet.main(args + ['--labeller', 'ann'])
+    with sync_playwright() as pw:
+        browser = pw.chromium.launch()
+        page = browser.new_page()
+        page.goto((tmp_path / 'yaw/ann/index.html').as_uri())
+        for key in ('1', '3', 'Backspace', '2', '4'):
+            page.keyboard.press(key)
+        text = page.locator('#csv').input_value()
+        progress = page.locator('#progress').inner_text()
+        browser.close()
+    rows = text.strip().split('\n')
+    expected = {f'{chosen[0]["imageId"]},{chosen[0]["faceId"]},profile',
+                f'{chosen[1]["imageId"]},{chosen[1]["faceId"]},frontal',
+                f'{chosen[2]["imageId"]},{chosen[2]["faceId"]},three-quarter',
+                f'{chosen[3]["imageId"]},{chosen[3]["faceId"]},unclear'}
+    assert rows[0] == 'imageId,faceId,yaw' and set(rows[1:]) == expected and rows[1:] == sorted(rows[1:])
+    assert progress.startswith('Face 4 of')