Add synthetic typed-scan vision fixtures and integration harness - #49
Merged
Conversation
Amends 0001's test-data tiers with a committed, deterministic vision-path fixture for the typed digital-export shape. Records why a clean typed scan is a real production input rather than the cautioned-against easy fake, and the (official, position, successful) scoring contract for the integration harness. Refs #48
Generator renders the Faker-seeded form to a flattened image-only PDF (no widgets, no text layer) that routes to the vision path, and emits an exact golden.json. Fills the columns the form-field fixture leaves blank (times_worked, mentor, level, successful) with a deterministic mix: times_worked deliberately differs from lane to expose column bleed, and the successful field covers true (typed initials), false (struck-through row), and null (mentor assigned, initials blank). Refs #48
Add the integration test that runs the real vision model over the committed flattened typed-scan and scores it against the exact golden on the (official, position, successful) tuple contract. The harness self-manages the Ollama daemon via OllamaDaemon (one `pytest -m integration` command — no manual `ollama serve` step) and adds --vision-model / --pull-models pytest options; a missing model skips with a copy-paste pull hint rather than silently downloading multiple GB. ACCURACY_FLOOR is 1.0: this is the easiest possible input (typed, clean, no handwriting) and humans clear even the hardest handwritten forms at ~99%, so anything short of a perfect read is a quality bug. The tests are expected to fail until those issues are fixed — that is their purpose. Tracked on #48. Refs #48
gavinbee
commented
May 30, 2026
| @@ -0,0 +1,390 @@ | |||
| """Regenerate the synthetic typed-scan vision fixture + its golden. | |||
Collaborator
Author
There was a problem hiding this comment.
Fine for this initial PR. However, we do need to turn this into a config driven synthetic scan maker. The hardcoding of all the data that ends up in the PDF makes me cry a bit on the inside.
Collaborator
Author
There was a problem hiding this comment.
Agreed — the hardcoding is bootstrap-only. Filed #50 to make the generator config-driven so this isn't lost. Keeping it out of this PR as you suggested.
Collaborator
Author
There was a problem hiding this comment.
Our input to make_synthetic_scan.py should be something like these goldens, plus a few more bits about how we want each row to look ... not in this pr though.
Collaborator
Author
There was a problem hiding this comment.
Captured this exactly in #50: the generator's input should be golden-shaped (rows + header) plus a few per-row rendering hints, with the golden emitted as the echoed-back source of truth. Not in this PR.
…scan Address PR review on #49: - Copy the blank Swim Ontario eval_form.pdf into the repo at tests/fixtures/templates/swim_ontario_v1/eval_form.pdf (one per supported template) so the fixture regenerates without an external eval-gen clone, and a material template change shows up as a diff to its committed blank. The vision generator now prefers this in-tree copy, falling back to a sibling eval-gen checkout. - Render officiating levels as Arabic numerals (2-5), matching how they appear on real forms, instead of Roman numerals. Regenerate the PDF and golden accordingly. Refs #48
Collaborator
Author
|
Addressed the review in 10463fd:
Default suite green (286 passed, 3 integration skipped). The level change doesn't touch the scored |
The blank template is a first-class reference asset, not a test fixture: nothing in the test suite opens it — only the fixture generators and (conceptually) template detection do. Move it out of tests/fixtures to a repo-root templates/<id>.pdf, flat-named by template id to match src/templates/<id>.py and docs/templates/. Point both the vision-scan and form-field generators at the shared in-tree copy (falling back to a sibling eval-gen checkout when refreshing it), so the canonical blank isn't used by only one generator. Refs #48
Follow-on to the template move: update the vision-scan and form-field generators to resolve the blank template at the repo-root templates/<id>.pdf, falling back to a sibling eval-gen checkout. Docstrings and --template help updated to match. Refs #48
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Lands the committable, deterministic vision-path integration fixture and the
pytest -m integrationharness for the typed digital-export workflow. This is meaningful, self-contained progress on #48 — it ships the fixtures, generator, golden, and harness. It does not close #48: making the harness pass is the remaining quality-iteration work, which stays on the issue.docs/design/0002-synthetic-vision-fixtures.md) — positions the typed-scan as a new tier alongside 0001's three (complements, doesn't replace, the real-scan and pseudonymized tiers).tests/fixtures/vision_scan/make_synthetic_scan.py) — renders the Faker-seeded form to a flattened, image-only PDF (no widgets, no text layer → routes to the vision path), with a deterministic mix of signed-off / struck-through / blank-initials rows, and emits an exact golden JSON.times_worked_positiondeliberately differs fromlane_numberto expose column bleed.session_1_evals_scan.pdf(2 pages) +session_1_evals_scan.golden.json.tests/test_integration_vision.py) — scores vision output on the(official, position, successful)tuple contract (modulo position-name normalization). Self-manages the Ollama daemon viaOllamaDaemon(one command, no manualollama serve), adds--vision-model/--pull-modelspytest options, and skips cleanly when Ollama or the model is absent.Quality gate (expected to fail)
ACCURACY_FLOOR = 1.0on purpose: this is the easiest possible input (typed, clean, no handwriting) and humans clear even the hardest handwritten forms at ~99%, so anything short of a perfect read is a quality bug. The integration tests are expected to fail until the issues they surface are fixed — that is their job. First run found: struck-through row read as signed-off, blank-initials+mentor row read as signed-off, a position vertical-bleed, a dropped row, and a doubled-token competition name (details in #48). These never run in CI (no GPU), so a red integration test blocks nothing automated.Test plan
pytest -q— default suite green (286 passed, 3 integration skipped).pytest -m integration— runs end-to-end against the real model, daemon auto-managed; currently red by design at the 100% floor.Refs #48
🤖 Generated with Claude Code