Synthetic typed-scan vision fixtures + integration harness

[gavinbee]

Goal

Add a committed, deterministic vision-path integration fixture for the typed digital-export workflow (a real input shape per the plan's Context: "a digital form is filled in and exported to PDF"), plus the pytest -m integration harness that scores vision output against its golden — and iterate on fixes/changes to the extraction pipeline until those tests pass.

This is a new tier alongside the three in 0001 §Test data acquisition. It complements — does not replace — Tier 2 (real handwritten scans + REMS goldens, local) and Tier 3 (pseudonymized realistic scans, deferred). It is explicitly easier than handwriting; its job is baseline-quality stabilization and regression-catching for straightforward cases, not handwriting robustness.

Scope

• docs/design/0002-synthetic-vision-fixtures.md — amends "Test data acquisition" to add the typed-scan tier and records why a clean typed scan is a legitimate production shape rather than the cautioned-against easy fake.
• Generator: render the Faker-seeded form to a flattened (no-widget) image-only PDF, fill the currently-blank fields (times_worked_position, mentor, level, successful) with a deterministic mix incl. signed-off / crossed-out / blank rows, and emit an exact golden JSON.
• Harness: pytest -m integration scores vision output on the documented (official, position, successful) tuple contract (modulo position-name normalization), skipped unless Ollama is present; self-manages the daemon via OllamaDaemon; --vision-model / --pull-models CLI options.
• Iterate until green. The accuracy bar for this easy typed input is 100% (humans clear even the hardest handwritten forms at ~99%, so tolerance here is essentially zero). The harness is expected to fail on first run — that is its job. This issue stays open and tracks the prompt/pipeline/model fixes needed to make it pass honestly. Do not lower the floor to go green. If a fix turns out to need a major design change, spin that out into its own issue + design doc.

Acceptance

pytest -m integration passes at ACCURACY_FLOOR = 1.0 on the committed synthetic scan, with the daemon auto-managed and no manual setup beyond an installed Ollama + pulled vision model.

Relationship to #16

Re-scopes #16's harness onto a committable synthetic golden instead of the rems-sync-dependent real-scan path. #16 stays open for the Tier-2 real-scan path.

[gavinbee]

First real-model run (qwen2.5vl:7b, 2-page synthetic scan)

Harness is wired and self-manages the daemon — pytest -m integration ran the full vision pipeline + scoring in ~90s with no manual setup. As expected at the 100% bar, it fails, surfacing concrete quality issues to fix:

test_tuple_contract_accuracy → 0.77 (17/22):

• Angela Carter — row dropped / name not matched.
• Daniel Adams — position Chief Timer, want Timer (likely vertical bleed from an adjacent "Chief Timer" row).
• James Mayo — successful read True, want False (struck-through row not honored — holistic-judgment miss).
• Rodney Trujillo — successful read True, want None (blank initials + mentor assigned read as signed off).

test_meet_header → competition_name came back 'Osborn Open Open 2026', want 'Osborn Open 2026' (doubled-token hallucination).

test_finds_every_official passes (10/11 matched by name).

Next: iterate on prompt/pipeline to close these. Decision recorded: keep ACCURACY_FLOOR = 1.0; these tests merge red and serve as the quality gate until fixed. Major fixes that need a new design doc get their own issue.

[gavinbee]

Handoff: picking up the quality iteration on another machine

The fixtures + harness half of this issue landed in PR #49 (squash 8a5afc4 on main). What remains is the expanded scope: iterate on the vision pipeline until the integration harness passes at its 100% accuracy floor. This comment is a self-contained brief for an agent starting cold on a different computer.

Current state (all on main)

• tests/test_integration_vision.py — the harness. 3 tests: test_finds_every_official, test_tuple_contract_accuracy, test_meet_header. Gated behind @pytest.mark.integration (skipped unless -m integration).
• tests/fixtures/vision_scan/make_synthetic_scan.py — deterministic generator (Faker seed 42) that flattens the blank form into an image-only PDF and writes a matching golden.
• tests/fixtures/vision_scan/session_1_evals_scan.pdf + session_1_evals_scan.golden.json — the committed fixture + golden (levels are Arabic 2–5).
• templates/swim_ontario_v1.pdf — the canonical blank Swim Ontario form (top-level, first-class asset; both generators resolve it via shared TEMPLATE_ID).
• tests/conftest.py — adds pytest options --vision-model <tag> (default vision_extract.DEFAULT_VISION_MODEL = qwen2.5vl:7b) and --pull-models (allow ollama pull of a missing model; off by default → skip with a hint).

How to run it

pytest -m integration tests/test_integration_vision.py
# optionally: --vision-model qwen2.5vl:7b   --pull-models

The parsed fixture self-manages the Ollama daemon via src.ollama_runtime.OllamaDaemon — it starts ollama serve only if one isn't already up and stops it on the way out. No manual ollama serve needed. On a box without Ollama / the model, the suite skips cleanly (with a ollama pull hint) rather than failing. Requires a GPU box with the model pulled to actually exercise.

The accuracy floor is deliberately 1.0 — do NOT lower it

This fixture is the easiest possible input: typed, clean, digital-export scan — no handwriting, skew, or smudges. Humans clear even the hardest handwritten forms at ~99%, so tolerance for error here is essentially zero. The tests are expected to fail until the underlying extraction-quality bugs are fixed — that is their job. The fix is in the model/prompt/pipeline, never in the threshold. (Rationale is documented at length in the module docstring and the ACCURACY_FLOOR comment.)

Known failures to close (representative — last real-model run, model output is not deterministic)

1. Struck-through row read as signed-off — e.g. James Mayo successful got True, want False. The model isn't treating a crossed-out row as not-successful.
2. Blank-initials + mentor-present row — e.g. Rodney Trujillo successful got True, want None. Should be ambiguous/null, not a positive sign-off.
3. Position vertical bleed — e.g. Daniel Adams position got Chief Timer, want Timer. Adjacent-row label bleeding into the read.
4. Dropped row — e.g. Angela Carter not found in output at all.
5. Doubled token in meet header — competition_name got Osborn Open Open 2026, want Osborn Open 2026.

The successful contract (whole-row holistic judgment → true/false/null) lives in src/vision_extract.py (_BASE_INSTRUCTIONS, build_prompt, _build_json_schema, _parse_successful). That's the first place to iterate for failures 1–2. Failures 3–5 are read/parse quality (prompt + JSON-schema constraints).

Workflow notes

• Per CONTRIBUTING: this gets its own branch off main (e.g. 48-improve-vision-quality), one PR referencing the issue. Imperative commit messages, no conventional-commit prefixes, Co-Authored-By trailer.
• The venv is Python 3.14 (.venv/Scripts/python.exe); project targets 3.12.
• Default (non-integration) suite must stay green and runs with Ollama not running — all LLM/subprocess calls are mocked by the conftest autouse fixtures.
• Related deferred work in Make the synthetic scan generator config-driven (golden-as-input) #50: turn make_synthetic_scan.py into a config-driven generator whose input is golden-shaped (rows + header config). Not part of this issue, but the two interact — keep the generator's golden output stable so the harness fixture doesn't churn.