[shashwat-deep]

Check E3 of protocol part E: how far each saved copy is from its original under the perceptual hashes reverse image search uses. Stacked on #15; merge that first.

What is in it

• eval/pdq.py: PDQ, Meta's 256-bit perceptual hash, in numpy. It is written from the algorithm of Meta's reference implementation (ThreatExchange pdq/cpp, BSD), not copied from it, and repeats its float32 arithmetic step for step:

  • luma;
  • two passes of the running-sum box filter along rows, then columns;
  • 64 × 64 samples;
  • a 16 × 16 DCT without the DC terms;
  • one bit per coefficient, set above the 128th smallest;
  • the quality metric.

  It reproduces the hashes Meta's C++ reference publishes for its eight regression images exactly: distance 0, quality 100. eval/tests/pdq_reference.csv pins those images to a ThreatExchange commit by SHA-256, and a test downloads them into data/ and checks every hash. It skips when offline.

• eval/render_copies.py DETECTIONS --pad P --threshold T --page URL saves every report image twice through the eval page's own render() (contract §3): with its masks at the operating point, and without masks as the control. The copies stay in the git-ignored data/rendered/, with a render_info.json of masks, SHA-256s and versions.

• eval/hash_distance.py data/rendered/<id> reports:

  • the share of masked copies within PDQ distance 31 of their original, with the bootstrap interval over images and broken down by masked share;
  • the same for the plain copies;
  • the full distributions of PDQ and pHash (imagehash.phash) distances;
  • how many originals have a PDQ quality of 49 or less.

  The JSON and CSV it writes hold distances, qualities and shares only, never hashes or pixels.

• Review guard. The page handling of run_detect.py moves into an EvalPage class that both scripts use. With a real eval page, they now refuse any image that review.csv doesn't keep, because part B runs no detector on an unreviewed image. The stub runs no detector, so it isn't affected. This covers part of what @06pratyush raised on Eval: check E2, saved files segment by segment #15 and Eval: yaw labelling page and Cohen's kappa #13.

• 8 new tests (101 eval tests pass). On a made-up textured photo with 8.9% of it masked, the masked copies came out at PDQ distance 28 to 30 from the original, still within 31, against 2 for the plain copies. That is the README's warning that a reverse image search may still match a covered copy.

For both of us to agree

Part E of the protocol now states:

• the original is the image as Chromium decodes it, upright and in sRGB;
• PDQ shrinks each picture to at most 512 px a side, keeping its shape, as Meta's Python reference does. The C++ tool instead stretches to exactly 512 × 512, which is why Meta's conformance rule allows distance ≤ 10 between implementations on their own image loading;
• the masked-share bins: none, under 1%, 1–5%, 5–10%, 10–25%, and 25% and over;
• originals with a PDQ quality of 49 or less are counted and reported, not dropped.

New eval-only dependencies: ImageHash 4.3.2 (BSD-2-Clause), scipy 1.18.1 (BSD-3-Clause), PyWavelets 1.10.0 (MIT and BSD-3-Clause).

🤖 Generated with Claude Code

[06pratyush]

Approving. Besides reading the code, I ran:

• the eval tests at the top of the stack on Windows: 104 passed, none skipped, so the PDQ reference test downloaded Meta's eight images and matched all of them;
• render_copies.py and hash_distance.py against the real eval page from #20, on a two-image stand-in set (the public-domain test photo, upright and stored turned 90°), at padding 0.10 and threshold 0.50. Each copy got one mask, rounded outward as contracts §2 says, covering 9.5% of the picture. The masked copies are at PDQ distance 44 from their originals and pHash distance 14; the plain copies at 0 and 0. The turned photo gives the same distances.

Two notes, neither blocking:

1. The E3 sentence in part E says more than the test checks. It reads as if eval/pdq.py gives the C++ hashes after shrinking each picture to at most 512 px a side, keeping its shape. test_pdq_matches_metas_reference_hashes never runs that shrink. It repeats the C++ tool's resize instead: it stretches each image with a side over 512 px to 512 × 512 by nearest neighbour, then calls hash_luma. E3 calls hash_image, which shrinks with Image.thumbnail((512, 512)), the call Meta's Python reference makes. The two paths differ:

• Meta's bridge-1-original.jpg (1600 × 1004 px) becomes 512 × 321 by thumbnail, against 512 × 512 in the C++ tool.
• Meta's own Python test expects hashes 4 to 8 bits from the C++ ones on these eight images, and allows up to 16.
• test_hash_image_shrinks_to_512_keeping_the_shape compares hash_image only with the same thumbnail call.
• All 800 report images in the manifest have a side over 512 px, so E3 always takes the path that no reference checks.

Suggested wording:

PDQ comes from eval/pdq.py, which shrinks each picture to at most 512 px a side, keeping its shape, as Meta's Python reference does. On the eight regression images of Meta's C++ reference, stretched to 512 × 512 by nearest neighbour as that reference does, it gives the same hashes (distance 0); the shrink itself is not compared with a reference.

To check the shrink as well, hash_image could be held to the hashes Meta's Python test expects for the same eight images.

2. render_copies.py renders only the images of the split that have a none line in the detections file. It stops only if none of them has one, and gives no warning for the rest.

• With a detections file from run_detect.py --split report --limit 100, E3 would cover 100 of the 800 report images.
• With --limit 100 and the default --split all, it would cover 75.
• render_info.json and the E3 report would still say "split": "report", and only the image count would show the gap.
• The run's run_info.json records the limit, but render_copies.py doesn't read it, and the run line has neither the split nor the limit.
• score.py reports imagesInSplit beside its count; E3 has no such field.

render_copies.py could stop unless every image of the split has a none line, and say how many are missing. That would also catch a detections file made on an older manifest: render_copies.py skips the run line, so it never compares the run's manifestSha256 with the manifest's.