[shashwat-deep]

First of the privacy checks (protocol part E): E2, the structure of saved files. Stacked on #14, which is stacked on #13, #12, #10, #9 and #2; merge those first.

What is in it

• eval/check_files.py FILES_OR_FOLDERS… [--out report.json] parses every saved JPEG and PNG with a parser of its own, independent of the app's stripJpeg and of the privacy suite's JPEG helper. It applies the protocol's E2 rule:

  • no APP1 (EXIF, XMP), APP11 (C2PA, JUMBF) or APP13 (IPTC) segment;
  • no thumbnail (JFIF thumbnail bytes or a JFXX extension);
  • an ICC profile only if every file checked in the run has the very same one. A profile that only some files carry fails too, since it came from the input;
  • nothing after the end-of-image marker.

  For PNG, the same rule rules out eXIf, tEXt, zTXt, iTXt, tIME and caBX (C2PA) chunks and anything after IEND, and chunk CRCs are checked. Other segments (a comment, an Adobe or MPF segment) are listed for us to judge. A broken file fails without stopping the run, and the script exits with an error if any file fails. The report holds segment names and sizes, never pixels or metadata values.

• 13 tests on files made to carry each kind of hidden data (91 eval tests in total).

• I saved three images from the draw through the deployed app on GitHub Pages in headless Chromium and checked the copies: all three pass, with segments JFIF, DQT, DQT, SOF0, DHT ×4, SOS, no ICC profile and nothing after the end.

  • Those three images had not been reviewed. Running the app on them went against part B, which says one of us looks at every sampled image before any detector runs on it. The copies were kept only in a temporary folder and no longer exist. Until the review is done, checks like this use synthetic images or our own photos.

After Pratyush's review

• The tests name files with Path(...).name, so they pass on Windows (15 E2 tests, 93 eval tests).
• A JFIF segment passes only at exactly the 14 bytes of a plain header. Anything longer, where bytes could hide, or shorter fails, as in the privacy suite.

For both of us

• Part E of the protocol now spells out the PNG equivalents of the rule, and that the ICC comparison runs over all the saved files of a run together.
• E2 also covers "every saved file … from the privacy suite". @06pratyush, the suite currently keeps its saved copies only in Playwright's temporary downloads. Keeping them, for example under tests/test-results/, would let us run check_files.py on them in CI. That's your call, since tests/ is yours.

E1 (exiftool on our own phone photos) needs exiftool, which isn't installed on my machine yet. E3 (perceptual-hash distance) comes next.

🤖 Generated with Claude Code

[06pratyush]

Two fixes before this can merge, and one question. #2 to #14 are merged; this PR now sits on main.

1. 12 of the 13 new tests fail on Windows (79 passed, 12 failed for pytest eval/tests here). eval/tests/test_check_files.py:35 indexes the report by e['file'].rsplit('/', 1)[-1], but on Windows str(path) uses backslashes, so every lookup ('plain.png', 'exif.jpg', …) raises KeyError. Path(e['file']).name fixes it; the checker itself is fine. CI runs only the app's tests, which is why this didn't show up there.
2. The JFIF check lets extra bytes through. jpeg_name() marks an APP0 JFIF segment clean when its thumbnail size bytes are 0, whatever the segment's length, so bytes appended inside the JFIF header pass E2. A clean JFIF header is exactly 14 bytes plus 3 per thumbnail pixel; failing any other length closes this. The privacy suite in tests/ checks it the same way.
3. Question. The description says three test images were saved through the deployed app, one of them stored turned, so they came from the Open Images draw. Part B says one author looks at every sampled image "before any detector runs on it", and eval/manifest/review.csv is still empty. Were those three reviewed first? If not, please note it, and let's use synthetic images or our own photos for checks like this until the review is done.

🤖 Review written with Claude Code

[06pratyush]

The fix does what I asked: the JFIF segment must now be exactly 14 bytes, and the new cases cover one with bytes after it (jfif-extra.jpg) and one cut short (jfif-short.jpg). The eval tests pass on Windows at the top of the stack (104, none skipped). Approving.