diff --git a/docs/design/0002-synthetic-vision-fixtures.md b/docs/design/0002-synthetic-vision-fixtures.md new file mode 100644 index 0000000..af31f8d --- /dev/null +++ b/docs/design/0002-synthetic-vision-fixtures.md @@ -0,0 +1,93 @@ +# 0002 — Synthetic typed-scan vision fixtures + integration harness + +Status: proposed + +## Context + +[0001 §Test data acquisition](0001-initial-design.md#test-data-acquisition) defines three tiers of test data: + +- **Tier 1** — committed *fillable* PDFs. These exercise the **form-field fast path** only; the vision model is never invoked. +- **Tier 2** — real scans + REMS golden sets, kept **local** (PII), run under `pytest -m integration`. This is the only tier that scores the **vision path**, and it depends on the rems-sync hookup ([#19]) plus the author manually collecting scans. +- **Tier 3** — pseudonymized, *realistic* scans (committed, deferred). It carries a deliberate warning: clean overlays "would erase exactly the artifacts our parser must learn to handle and make the fixtures **trivially easier than real scans**." + +The gap this doc closes: **there is no committed, deterministic fixture that exercises the vision path.** Today the only thing driving vision extraction is an ad-hoc `data/sample_scan.pdf` with no committed ground truth, so we cannot score the model, catch regressions, or stabilize quality on straightforward cases. gh [#16] (the golden integration test harness) is written against Tier 2, which can't land until rems-sync is wired and the author has hand-collected scans — and even then it can never be committed. + +We want a committed, deterministic, vision-path integration fixture **now**, for the straightforward (typed) case, without waiting on Tier 2 or Tier 3. + +## Why this isn't the "trivially easier fake" Tier 3 warns against + +The Tier 3 caution is specifically about **faking handwriting** — drawing clean fonts where a real form would have messy ink, thereby erasing the artifacts the parser must learn to handle. A clean, typed, flattened PDF is a different thing: it is a **real production input shape**, not a surrogate for a handwritten one. + +0001's own Context lists it explicitly: + +> "sometimes (e.g. Swim Ontario's online form option) a digital form is filled in and **exported to PDF**" + +A digital-export PDF that has been flattened (or printed-then-scanned-to-image) is typed text with no widget layer and no text layer — exactly the shape that routes to the vision path (`has_form_fields() == False`, no embedded text). It is genuinely *easier* than a handwritten scan, and that is the point: this tier's job is **baseline-quality stabilization and regression-catching for straightforward cases**, not handwriting robustness. Handwriting robustness stays the job of Tier 2 (real scans) and Tier 3 (pseudonymized realistic scans), both unchanged by this doc. + +So this is an **addition** to the tier list, not a replacement: + +| Tier | Path exercised | Committed? | Difficulty | Status | +|---|---|---|---|---| +| 1 | form-field | yes | n/a | implemented | +| **1b (this doc)** | **vision** | **yes** | **typed / clean** | **proposed** | +| 2 | vision | no (local) | real handwriting | gh #16, blocked on #19 | +| 3 | vision | yes (deferred) | realistic handwriting | gh #29 | + +## Design + +### Generator — `tests/fixtures/vision_scan/make_synthetic_scan.py` + +Mirrors the existing `tests/fixtures/form_field/make_synthetic_fixture.py` (same Faker seed → reproducible), but produces a **flattened, image-only** PDF and an **exact golden** instead of a fillable PDF. + +1. **Fill the form** from the same constrained vocabularies and Faker-seeded names the form-field generator uses, so the two fixtures stay recognizably the same meet shape. +2. **Fill the currently-blank fields too.** The form-field fixture leaves `times_worked_position`, `mentor`, `level`, and `successful` blank. This tier fills them with a deterministic pattern so the harness actually tests the interesting columns (the ones where the current model misreads — see Open items) and the holistic `successful` judgement. +3. **Flatten + rasterize.** Use PyMuPDF `doc.bake()` to flatten the AcroForm widgets into static page content, then rasterize each page to a PNG and rebuild a PDF whose pages are *only* that image. The result has no widgets and no text layer, guaranteeing it routes to the vision path (matches the observed `data/sample_scan.pdf`: `widgets=0, text_chars=0, images=1`). +4. **Emit the golden** (`golden.json`) from the same known values — no model in the loop, so the golden is exact by construction. + +### Modelling `successful` honestly for a typed form + +A typed digital-export form has no checkbox for "successful" — it has an initials cell. So the two *natural* typed states are: + +- **initials present → `true`** (signed off) +- **blank cell → `null`** (not signed off / ambiguous) + +To also exercise the holistic **`false`** judgement (and the "blank initials but a mentor was assigned" ambiguity the schema cares about), the generator deterministically draws a small number of **marks** onto the rasterized image: e.g. one row struck through end-to-end (clear `false`), and one row with a mentor filled but initials blank (genuine `null`). These are drawn programmatically with known coordinates, so the golden records the intended verdict exactly. We keep these marks minimal and clean — rich, realistic ink for these cases remains Tier 3's job; here they exist only so every `successful` branch (`true`/`false`/`null`) appears at least once. + +### Golden format and the scoring contract + +The golden reuses the comparison contract 0001 already specifies in Verification step 13: + +> parser output matches the golden set on **`(official, position, successful)`** tuples … **modulo position-name normalization**. + +`golden.json` therefore records, per row: `official_name`, `position`, `successful`, plus the meet/session header for a coarser header check. The full Faker-known values are also recorded so a future, stricter comparison can opt in without regenerating. + +### Harness — `tests/test_integration_vision.py` + +- Marked `@pytest.mark.integration`; **skipped unless `-m integration`** is passed (CI stays vision-free, per 0001 Verification 14). Also skips with a clear message if Ollama / the model isn't reachable, so `-m integration` on a machine without a GPU degrades to a skip rather than a failure. +- Runs the real vision path over the committed synthetic scan, then scores against `golden.json`: + - **Row matching** by `official_name` (normalized: case/space-folded). + - **Per-tuple assertions** on `(official, position, successful)` with **position-name normalization** (a small synonym/canonicalization map, since the model may return "Inspector of Turns" vs "Turn Inspector" etc.). + - **Accuracy threshold, not exact match.** Vision output is non-deterministic even at temperature 0 across model/runtime versions, so the harness asserts a field-level accuracy floor (e.g. ≥ 0.9 of scored tuples correct) and reports the mismatches, rather than requiring a perfect transcription. The threshold is a named constant, tunable as we learn the model's real hit rate on this fixture. + +### `pytest.ini` / markers + +Register the `integration` marker (if not already) so `-m integration` is first-class and unmarked runs skip it cleanly. + +## Verification + +1. `python tests/fixtures/vision_scan/make_synthetic_scan.py` regenerates an identical PDF + golden for a fixed seed. +2. The generated PDF reports `has_form_fields() == False` and zero embedded text — i.e. it routes to the vision path. +3. `pytest tests/ -q` (no marker) still passes with **no** Ollama and **skips** the new integration test. +4. `pytest -m integration` on a machine with Ollama + the model runs the vision path against the committed scan and passes the accuracy threshold; with Ollama absent it **skips** with a clear message. +5. The golden's `(official, position, successful)` tuples match the Faker-known values by construction (a cheap unit test can assert the golden against the generator's in-memory data without any model). + +## Open items / out of scope + +- **Handwriting realism stays Tier 3** (gh #29). This tier is clean/typed by design. +- **Quality bugs surfaced while scoping**, to be filed once the harness can measure them: (a) `times_worked_position` taking the `lane_number` value (column bleed); (b) hallucinated `successful` rationales ("Row is crossed out" on blank cells). Both are likely prompt-level fixes and the harness is what lets us prove a fix helps. +- **Threshold tuning.** The initial accuracy floor is a guess; we adjust it against observed hit rates once the fixture exists. +- **gh #16 stays open** for the Tier-2 real-scan path; this doc re-scopes the *committable* portion of its harness onto the synthetic golden. + +[#16]: https://github.com/swimblocks/deck-eval-parser/issues/16 +[#19]: https://github.com/swimblocks/deck-eval-parser/issues/19 +[#29]: https://github.com/swimblocks/deck-eval-parser/issues/29 diff --git a/docs/design/README.md b/docs/design/README.md index 1e76441..994cba9 100644 --- a/docs/design/README.md +++ b/docs/design/README.md @@ -5,3 +5,4 @@ Sequential design docs for non-trivial features. See [CONTRIBUTING.md](../../CON | # | Title | Status | |---|---|---| | [0001](0001-initial-design.md) | Initial design — single-PDF parser, local vision LLM, JSON-canonical output, template registry, agentic interactive review | implemented (in progress) | +| [0002](0002-synthetic-vision-fixtures.md) | Synthetic typed-scan vision fixtures + integration harness — committed, deterministic vision-path golden for the typed digital-export shape | proposed | diff --git a/pytest.ini b/pytest.ini new file mode 100644 index 0000000..6ea5a82 --- /dev/null +++ b/pytest.ini @@ -0,0 +1,4 @@ +[pytest] +testpaths = tests +markers = + integration: end-to-end test that invokes the real vision model via Ollama. Skipped unless `pytest -m integration` is passed (see CONTRIBUTING.md). diff --git a/templates/swim_ontario_v1.pdf b/templates/swim_ontario_v1.pdf new file mode 100644 index 0000000..aa04188 Binary files /dev/null and b/templates/swim_ontario_v1.pdf differ diff --git a/tests/conftest.py b/tests/conftest.py index 0db2025..0d6bb1f 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -1,6 +1,52 @@ -"""Shared pytest fixtures. +"""Shared pytest fixtures and collection hooks. -For now this only houses the bookkeeping shape; the Ollama HTTP-client -stub and subprocess-spawn stub land alongside their respective modules +For now this houses the bookkeeping shape; the Ollama HTTP-client stub and +subprocess-spawn stub land alongside their respective modules (``src.ollama_runtime``) so that test_*.py files can opt in or out. + +It also gates ``@pytest.mark.integration`` tests: they invoke the real +vision model via Ollama and are **skipped unless ``pytest -m integration`` +is passed** (see CONTRIBUTING.md). A plain ``pytest`` run — including CI — +collects them but skips, so the default suite never needs a GPU or a +running daemon. """ +import pytest + + +def pytest_addoption(parser): + """CLI options for the integration suite (see test_integration_vision).""" + group = parser.getgroup("integration") + group.addoption( + "--vision-model", + action="store", + default=None, + help="Vision model tag for integration tests " + "(default: vision_extract.DEFAULT_VISION_MODEL).", + ) + group.addoption( + "--pull-models", + action="store_true", + default=False, + help="Allow integration tests to `ollama pull` a missing model " + "(a multi-GB download). Off by default: a missing model skips " + "with a copy-paste pull hint instead.", + ) + + +def pytest_collection_modifyitems(config, items): + """Skip integration tests unless ``-m integration`` was requested. + + We key off the ``-m`` marker expression rather than inventing a custom + flag so the documented invocation (``pytest -m integration``) is the + single gate. When the user explicitly selects integration tests, pytest + already deselects everything else, so we leave the items untouched. + """ + markexpr = config.option.markexpr or "" + if "integration" in markexpr: + return + skip_integration = pytest.mark.skip( + reason="integration test; run with `pytest -m integration`" + ) + for item in items: + if "integration" in item.keywords: + item.add_marker(skip_integration) diff --git a/tests/fixtures/form_field/make_synthetic_fixture.py b/tests/fixtures/form_field/make_synthetic_fixture.py index 354061f..c52c6cd 100644 --- a/tests/fixtures/form_field/make_synthetic_fixture.py +++ b/tests/fixtures/form_field/make_synthetic_fixture.py @@ -21,9 +21,10 @@ ``tests/test_form_extract.py`` that reference specific synthetic names. Inputs: - The blank Swim Ontario eval_form.pdf, located by default at - ``../../eval-gen/eval_form.pdf`` relative to this repo. Swim Ontario - publishes the form publicly; the blank contains no PII. + The blank Swim Ontario template, committed in-tree at + ``templates/swim_ontario_v1.pdf`` (falling back to a sibling eval-gen + checkout when refreshing it). Swim Ontario publishes the form publicly; + the blank contains no PII. Outputs: Overwrites ``session_1_evals.pdf`` in this directory. @@ -200,19 +201,30 @@ def generate( output.close() +# Template id this fixture is generated for. The blank form lives in-tree +# at the repo-root templates/.pdf (a first-class reference asset shared +# with the vision-scan generator). +TEMPLATE_ID = "swim_ontario_v1" + + def _default_template_path() -> Path: - """Best-effort guess at where the blank eval_form.pdf lives locally.""" - candidates = [ - Path(__file__).resolve().parents[3].parent / "eval-gen" / "eval_form.pdf", + """Locate the blank template, preferring the committed in-tree copy.""" + repo_root = Path(__file__).resolve().parents[3] + in_tree = repo_root / "templates" / f"{TEMPLATE_ID}.pdf" + if in_tree.is_file(): + return in_tree + # Fall back to a sibling eval-gen checkout (handy when refreshing the + # committed blank from a new export). + for c in ( + repo_root.parent / "eval-gen" / "eval_form.pdf", Path("C:/Users/gavbe/src/eval-gen/eval_form.pdf"), - ] - for c in candidates: + ): if c.is_file(): return c raise FileNotFoundError( - "Could not find eval-gen/eval_form.pdf. Either clone " - "https://github.com/swimblocks/deck-eval-gen alongside this repo, or " - "pass --template /path/to/eval_form.pdf" + f"Could not find a blank template. Expected the committed copy at " + f"{in_tree}, or a sibling eval-gen checkout. Pass --template " + f"/path/to/eval_form.pdf to override." ) @@ -225,8 +237,8 @@ def _parse_args() -> argparse.Namespace: "--template", type=Path, default=None, - help="Path to the blank Swim Ontario eval_form.pdf. " - "Defaults to eval-gen/eval_form.pdf next to this repo.", + help="Path to the blank Swim Ontario template. " + "Defaults to the in-tree templates/swim_ontario_v1.pdf.", ) p.add_argument( "--output", diff --git a/tests/fixtures/vision_scan/make_synthetic_scan.py b/tests/fixtures/vision_scan/make_synthetic_scan.py new file mode 100644 index 0000000..4160fbc --- /dev/null +++ b/tests/fixtures/vision_scan/make_synthetic_scan.py @@ -0,0 +1,416 @@ +"""Regenerate the synthetic typed-scan vision fixture + its golden. + +This is the **vision-path** sibling of +``tests/fixtures/form_field/make_synthetic_fixture.py``. Where that script +produces a *fillable* PDF (exercising the form-field fast path), this one +produces a **flattened, image-only** PDF — no widgets, no text layer — that +routes through the vision model, plus an exact ``golden.json`` describing +what's on it. + +The point (see ``docs/design/0002-synthetic-vision-fixtures.md``): a typed +digital-export form that's been flattened is a *real production input +shape*, and it doubles as a committed, deterministic regression fixture for +the straightforward (typed) case. It is intentionally easier than a +handwritten scan — handwriting robustness stays the job of the Tier-2 (real +scans) and Tier-3 (pseudonymized) fixtures. + +Everything is generated from a fixed Faker seed, so re-running with the same +seed and Faker version reproduces an identical PDF + golden. No real +person's information is ever committed. + +Recorded command for the committed fixture: + +.. code-block:: bash + + python tests/fixtures/vision_scan/make_synthetic_scan.py --seed 42 + +Inputs: + A blank template PDF, committed in-tree at + ``templates/.pdf`` (one per supported template, alongside + ``src/templates/.py`` and ``docs/templates/``). It's a + first-class reference asset, not a test fixture: nothing in the test + suite opens it — only this generator and template detection do — so it + lives at the repo root rather than under ``tests/``. Keeping a copy in + the source tree means anyone can regenerate the fixture without cloning + eval-gen, and a material change to a province's form shows up as a diff + to its committed blank — a useful signal when template *detection* + starts failing on a redesigned form. ``--template`` overrides the path + (e.g. to regenerate from a fresh eval-gen export). + +Outputs (overwritten in this directory): + ``session_1_evals_scan.pdf`` — flattened image-only PDF + ``session_1_evals_scan.golden.json`` — exact ground truth +""" +from __future__ import annotations + +import argparse +import io +import json +import re +import sys +from itertools import cycle +from pathlib import Path + +import fitz +from faker import Faker +from PIL import Image, ImageDraw + + +# --------------------------------------------------------------------------- +# Constrained vocabularies — kept in step with the form-field generator so +# both fixtures read as the same kind of meet. +# --------------------------------------------------------------------------- + +POSITIONS = [ + "Starter", + "Stroke Judge", + "Inspector of Turns", + "Inspector of Turns", + "Chief Timer", + "Timer", + "Timer", + "Admin Desk", + "Recorder", + "Timer", + "Timer", +] + +LANE_BY_POSITION = { + "Inspector of Turns": cycle(["4", "5", "6", "7"]), + "Timer": cycle(["1", "2", "3", "4", "5", "6", "7", "8"]), +} + +# The mentor's officiating level. On these forms the level is virtually +# always written as an Arabic numeral, so the fixture uses Arabic too. +LEVELS = cycle(["2", "3", "4", "5"]) + +# How many times the official has worked the position — a small spread of +# plausible values. Deliberately *not* equal to the lane number, so the +# harness can catch the model copying the lane column into this one +# (the "column bleed" bug noted in design doc 0002). +TIMES_WORKED = cycle(["1", "3", "5", "8", "12", "20"]) + +ROWS_PER_PAGE = 9 +DUPLICATE_SUFFIX_RE = re.compile(r"\s\[\d+\]$") + +# Rasterization DPI. Matches src/pdf_io.DEFAULT_DPI so the committed scan +# resembles what the parser would render from a real flattened PDF. +DEFAULT_DPI = 200 + + +# --------------------------------------------------------------------------- +# successful-intent scheme (deterministic) +# +# A typed form's natural states are "initials present -> true" and +# "blank -> null". To exercise a clear ``false`` and a genuine ``null`` we +# pick two specific 0-based official indices and render them specially: +# - FALSE_INDEX: whole row struck through (clear not-successful). +# - NULL_INDEX: mentor assigned but initials left blank (ambiguous). +# Every other row gets typed initials -> true. +# --------------------------------------------------------------------------- + +FALSE_INDEX = 7 # page 1, row 8 +NULL_INDEX = 8 # page 1, row 9 + + +def _initials(name: str) -> str: + parts = [p for p in re.split(r"\s+", name.strip()) if p] + return "".join(p[0].upper() for p in parts[:2]) or "X" + + +def _make_officials(faker: Faker, n: int) -> list[dict]: + out: list[dict] = [] + for i in range(n): + position = POSITIONS[i % len(POSITIONS)] + lane_iter = LANE_BY_POSITION.get(position) + lane = next(lane_iter) if lane_iter is not None else "" + mentor = faker.name() + + if i == FALSE_INDEX: + successful = False + initials = "" # struck-through row; no sign-off initials + elif i == NULL_INDEX: + successful = None + initials = "" # mentor assigned, but not signed off + else: + successful = True + initials = _initials(mentor) + + out.append({ + "name": faker.name(), + "club": faker.bothify(text="???", letters="ABCDEFGHIJKLMNOPQRSTUVWXYZ"), + "position": position, + "lane": lane, + "times": next(TIMES_WORKED), + "mentor": mentor, + "level": next(LEVELS), + "initials": initials, + "successful": successful, + }) + return out + + +def _meet_header(faker: Faker, total_pages: int) -> dict[str, str]: + return { + "Competition Name": f"{faker.last_name()} Open 2026", + "Competition Coordinator": faker.name(), + "Level": "", # cc_level intentionally blank + "Date Session": "Fri, Jan 9, 2026 / Session 1", + "Host Club": faker.bothify(text="???", letters="ABCDEFGHIJKLMNOPQRSTUVWXYZ"), + "COC": faker.name(), + "Page": "", # set per page below + "of": str(total_pages), + } + + +def _row_widget_values(o: dict) -> dict[str, str]: + """Map base widget names -> typed value for one official's row.""" + return { + "Name of OfficialRow": o["name"], + "ClubRow": o["club"], + "PositionRow": o["position"], + "Lane numberRow": o["lane"], + "How many times have you worked this positionRow": o["times"], + "Mentor Official Session refereeRow": o["mentor"], + "LevelRow": o["level"], + "Successful initialRow": o["initials"], + } + + +def _blank_row_values() -> dict[str, str]: + return {k: "" for k in _row_widget_values( + {"name": "", "club": "", "position": "", "lane": "", "times": "", + "mentor": "", "level": "", "initials": ""} + )} + + +def _write_widget(widget, value: str) -> None: + widget.field_value = value + widget.update() + + +def _render_page_image( + template_path: Path, + page_meet: dict[str, str], + page_officials: list[dict], + dpi: int, +) -> Image.Image: + """Fill, flatten, and rasterize one page; return a PIL image. + + Strike-through marks for not-successful rows are drawn onto the + rasterized pixels using the (pre-bake) widget geometry, so they land + exactly over the intended row. + """ + template = fitz.open(template_path) + page = template[0] + + # y-center + x-extent (PDF points) of any row we need to strike through. + strike_rows: list[tuple[float, float, float]] = [] # (x0, x1, y_center) + + for widget in page.widgets() or (): + bare = DUPLICATE_SUFFIX_RE.sub("", widget.field_name) + + if bare in page_meet: + _write_widget(widget, page_meet[bare]) + continue + + m = re.match(r"^(.+?)Row(\d+)$", bare) + if not m: + continue + base, row_index = m.group(1), int(m.group(2)) + base_key = f"{base}Row" + + if row_index <= len(page_officials): + o = page_officials[row_index - 1] + row_vals = _row_widget_values(o) + # Capture geometry off the leftmost (name) widget for a row we + # render as struck-through. + if base_key == "Name of OfficialRow" and o["successful"] is False: + r = widget.rect + # Extend to the row's right edge (the Successful column). + strike_rows.append((r.x0, 736.2, (r.y0 + r.y1) / 2.0)) + else: + row_vals = _blank_row_values() + _write_widget(widget, row_vals.get(base_key, "")) + + # Flatten widgets into static page content, then rasterize. + template.bake(annots=True, widgets=True) + zoom = dpi / 72.0 + pix = page.get_pixmap(matrix=fitz.Matrix(zoom, zoom), alpha=False) + img = Image.frombytes("RGB", (pix.width, pix.height), pix.samples) + template.close() + + # Draw strike-throughs in pixel space. + draw = ImageDraw.Draw(img) + for x0, x1, yc in strike_rows: + draw.line( + [(x0 * zoom, yc * zoom), (x1 * zoom, yc * zoom)], + fill=(0, 0, 0), + width=max(2, round(zoom * 1.2)), + ) + + return img + + +def _build_golden( + output_pdf_name: str, + meet: dict[str, str], + officials: list[dict], + pages_of_officials: list[list[dict]], +) -> dict: + """Exact ground truth for the rendered scan (no model in the loop).""" + rows = [] + for page_index, page_officials in enumerate(pages_of_officials): + for row_index, o in enumerate(page_officials, start=1): + rows.append({ + "source_page": page_index + 1, + "row_index": row_index, + "official_name": o["name"], + "club": o["club"], + "position": o["position"], + "lane_number": o["lane"] or None, + "times_worked_position": o["times"], + "mentor": o["mentor"], + "level": o["level"], + "successful": o["successful"], + }) + return { + "source_pdf": output_pdf_name, + "template_id": "swim_ontario_v1", + "meet": { + "competition_name": meet["Competition Name"], + "host_club": meet["Host Club"], + "coc": meet["COC"], + }, + "session": { + "competition_coordinator": meet["Competition Coordinator"], + "cc_level": None, + "date_session": "Fri, Jan 9, 2026 / Session 1", + "session_number": 1, + }, + "rows": rows, + } + + +def generate( + template_path: Path, + output_pdf: Path, + output_golden: Path, + seed: int, + pages: int, + officials_on_last_page: int, + dpi: int, +) -> None: + faker = Faker("en_CA") + Faker.seed(seed) + + total_officials = (pages - 1) * ROWS_PER_PAGE + officials_on_last_page + officials = _make_officials(faker, total_officials) + meet = _meet_header(faker, total_pages=pages) + + pages_of_officials = [ + officials[i : i + ROWS_PER_PAGE] + for i in range(0, len(officials), ROWS_PER_PAGE) + ] + + out = fitz.open() + for page_index in range(pages): + page_officials = pages_of_officials[page_index] + page_meet = dict(meet) + page_meet["Page"] = str(page_index + 1) + + img = _render_page_image(template_path, page_meet, page_officials, dpi) + + # Insert the rendered page as a pure image — no widgets, no text + # layer — so the parser routes it to the vision path. + buf = io.BytesIO() + img.save(buf, format="PNG", optimize=True) + new_page = out.new_page(width=792, height=612) + new_page.insert_image(new_page.rect, stream=buf.getvalue()) + + out.save(output_pdf, deflate=True, garbage=4, clean=True) + out.close() + + golden = _build_golden(output_pdf.name, meet, officials, pages_of_officials) + output_golden.write_text( + json.dumps(golden, indent=2, ensure_ascii=False) + "\n", + encoding="utf-8", + ) + + +# Template id this fixture is generated for. The blank form lives in-tree +# at the repo-root templates/.pdf (a first-class reference asset). +TEMPLATE_ID = "swim_ontario_v1" + + +def _in_tree_template_path() -> Path: + """The committed blank template for ``TEMPLATE_ID`` (repo-root templates/).""" + repo_root = Path(__file__).resolve().parents[3] + return repo_root / "templates" / f"{TEMPLATE_ID}.pdf" + + +def _default_template_path() -> Path: + # Prefer the in-tree copy so regeneration needs no external clone. + in_tree = _in_tree_template_path() + if in_tree.is_file(): + return in_tree + # Fall back to a sibling eval-gen checkout (handy when refreshing the + # committed blank from a new export). + for c in ( + Path(__file__).resolve().parents[3].parent / "eval-gen" / "eval_form.pdf", + Path("C:/Users/gavbe/src/eval-gen/eval_form.pdf"), + ): + if c.is_file(): + return c + raise FileNotFoundError( + f"Could not find a blank template. Expected the committed copy at " + f"{in_tree}, or a sibling eval-gen checkout. Pass --template " + f"/path/to/eval_form.pdf to override." + ) + + +def _parse_args() -> argparse.Namespace: + p = argparse.ArgumentParser( + description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + p.add_argument("--template", type=Path, default=None, + help="Path to blank eval_form.pdf (defaults next to repo).") + p.add_argument("--output", type=Path, + default=Path(__file__).parent / "session_1_evals_scan.pdf", + help="Output path for the flattened image-only PDF.") + p.add_argument("--golden", type=Path, default=None, + help="Output path for the golden JSON " + "(defaults to .golden.json).") + p.add_argument("--seed", type=int, default=42, help="Faker seed.") + p.add_argument("--pages", type=int, default=2, help="Number of pages.") + p.add_argument("--officials-on-last-page", type=int, default=2, + help="How many of the 9 row slots on the last page to fill.") + p.add_argument("--dpi", type=int, default=DEFAULT_DPI, + help="Rasterization DPI.") + return p.parse_args() + + +def main() -> int: + args = _parse_args() + template = args.template or _default_template_path() + golden = args.golden or args.output.with_suffix(".golden.json") + print(f"Reading template: {template}") + print(f"Writing scan: {args.output}") + print(f"Writing golden: {golden}") + print(f"Seed: {args.seed}") + generate( + template_path=template, + output_pdf=args.output, + output_golden=golden, + seed=args.seed, + pages=args.pages, + officials_on_last_page=args.officials_on_last_page, + dpi=args.dpi, + ) + print("Done.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/fixtures/vision_scan/session_1_evals_scan.golden.json b/tests/fixtures/vision_scan/session_1_evals_scan.golden.json new file mode 100644 index 0000000..ab93abc --- /dev/null +++ b/tests/fixtures/vision_scan/session_1_evals_scan.golden.json @@ -0,0 +1,149 @@ +{ + "source_pdf": "session_1_evals_scan.pdf", + "template_id": "swim_ontario_v1", + "meet": { + "competition_name": "Osborn Open 2026", + "host_club": "DEU", + "coc": "Michael Carlson" + }, + "session": { + "competition_coordinator": "Johnny Campos", + "cc_level": null, + "date_session": "Fri, Jan 9, 2026 / Session 1", + "session_number": 1 + }, + "rows": [ + { + "source_page": 1, + "row_index": 1, + "official_name": "Noah Rhodes", + "club": "RCS", + "position": "Starter", + "lane_number": null, + "times_worked_position": "1", + "mentor": "Allison Hill", + "level": "2", + "successful": true + }, + { + "source_page": 1, + "row_index": 2, + "official_name": "Alyssa Gonzalez", + "club": "UWR", + "position": "Stroke Judge", + "lane_number": null, + "times_worked_position": "3", + "mentor": "Allen Robinson", + "level": "3", + "successful": true + }, + { + "source_page": 1, + "row_index": 3, + "official_name": "Olivia Moore", + "club": "NKI", + "position": "Inspector of Turns", + "lane_number": "4", + "times_worked_position": "5", + "mentor": "Jerry Ramirez", + "level": "4", + "successful": true + }, + { + "source_page": 1, + "row_index": 4, + "official_name": "Brent Abbott", + "club": "TIZ", + "position": "Inspector of Turns", + "lane_number": "5", + "times_worked_position": "8", + "mentor": "Tyler Rogers", + "level": "5", + "successful": true + }, + { + "source_page": 1, + "row_index": 5, + "official_name": "Angela Carter", + "club": "UTL", + "position": "Chief Timer", + "lane_number": null, + "times_worked_position": "12", + "mentor": "Joe Martinez", + "level": "2", + "successful": true + }, + { + "source_page": 1, + "row_index": 6, + "official_name": "Dylan Miller", + "club": "HDM", + "position": "Timer", + "lane_number": "1", + "times_worked_position": "20", + "mentor": "Michele Williams", + "level": "3", + "successful": true + }, + { + "source_page": 1, + "row_index": 7, + "official_name": "Daniel Adams", + "club": "VUC", + "position": "Timer", + "lane_number": "2", + "times_worked_position": "1", + "mentor": "Mark Diaz", + "level": "4", + "successful": true + }, + { + "source_page": 1, + "row_index": 8, + "official_name": "James Mayo", + "club": "UWR", + "position": "Admin Desk", + "lane_number": null, + "times_worked_position": "3", + "mentor": "Carolyn Daniel", + "level": "5", + "successful": false + }, + { + "source_page": 1, + "row_index": 9, + "official_name": "Rodney Trujillo", + "club": "MIC", + "position": "Recorder", + "lane_number": null, + "times_worked_position": "5", + "mentor": "Frederick Tate", + "level": "2", + "successful": null + }, + { + "source_page": 2, + "row_index": 1, + "official_name": "Matthew Foster", + "club": "UOE", + "position": "Timer", + "lane_number": "3", + "times_worked_position": "8", + "mentor": "Tommy Walter", + "level": "3", + "successful": true + }, + { + "source_page": 2, + "row_index": 2, + "official_name": "Kevin Hurst", + "club": "MLH", + "position": "Timer", + "lane_number": "4", + "times_worked_position": "12", + "mentor": "David Hoffman", + "level": "4", + "successful": true + } + ] +} diff --git a/tests/fixtures/vision_scan/session_1_evals_scan.pdf b/tests/fixtures/vision_scan/session_1_evals_scan.pdf new file mode 100644 index 0000000..0a4b643 Binary files /dev/null and b/tests/fixtures/vision_scan/session_1_evals_scan.pdf differ diff --git a/tests/test_integration_vision.py b/tests/test_integration_vision.py new file mode 100644 index 0000000..aafea94 --- /dev/null +++ b/tests/test_integration_vision.py @@ -0,0 +1,297 @@ +"""Vision-path integration test against the committed synthetic golden. + +This is the committable half of gh #16's golden harness (see +``docs/design/0002-synthetic-vision-fixtures.md``). It runs the **real** +vision model over ``tests/fixtures/vision_scan/session_1_evals_scan.pdf`` +— a flattened, image-only PDF generated deterministically from a Faker +seed — and scores the output against the exact ``golden.json`` emitted +alongside it. + +Gating: + * ``@pytest.mark.integration`` → skipped unless ``pytest -m integration`` + (conftest enforces this). CI never runs it. + * Skips with a clear message if Ollama isn't reachable or the model + isn't pulled, so ``-m integration`` on a GPU-less box degrades to a + skip rather than a hard failure. + +Scoring contract (from 0001 Verification step 13): match rows by official +name, then assert ``(official, position, successful)`` tuples agree — +**modulo position-name normalization** — at a 100% accuracy floor. This +fixture is the easiest possible input (typed, clean, no handwriting), and +humans clear even the hardest handwritten forms at ~99%, so anything short +of a perfect read here is a quality bug. These tests are therefore +**expected to fail until the extraction-quality issues they surface are +fixed** — that is their job. Mismatches are reported in the assertion +message so each failure points at the offending rows. + +Daemon lifecycle is self-managed: the ``parsed`` fixture drives the same +:class:`~src.ollama_runtime.OllamaDaemon` the CLI uses, so a single +``pytest -m integration`` is the whole story — it starts the daemon if +one isn't already up and stops it on the way out (leaving a +pre-existing daemon alone). No manual ``ollama serve`` two-step. + +CLI options (see ``tests/conftest.py``): + * ``--vision-model `` overrides the model (default + ``vision_extract.DEFAULT_VISION_MODEL``). + * ``--pull-models`` allows the test to ``ollama pull`` a missing + model (a multi-GB download). Off by default: a missing model skips + with a copy-paste pull hint instead of silently downloading. +""" +from __future__ import annotations + +import json +import re +from pathlib import Path + +import pytest + +from src import merge, ollama_runtime as rt, vision_extract +from src.templates import TEMPLATES + +pytestmark = pytest.mark.integration + + +FIXTURE_DIR = Path(__file__).parent / "fixtures" / "vision_scan" +SCAN_PDF = FIXTURE_DIR / "session_1_evals_scan.pdf" +GOLDEN_PATH = FIXTURE_DIR / "session_1_evals_scan.golden.json" + +TEMPLATE_ID = "swim_ontario_v1" + +# Accuracy floor for the (official, position, successful) tuple contract. +# +# This is deliberately 100%, not a "good enough" threshold. The fixture is +# the EASIEST possible input: a typed, clean, digital-export scan with no +# handwriting, no skew, no smudges. Humans transcribe even the *hardest* +# handwritten versions of these forms at ~99% accuracy, so our tolerance +# for error on the easy case is essentially zero — anything less than a +# perfect read here is a quality bug to fix, not slack to absorb. +# +# These tests are EXPECTED to fail until the underlying extraction quality +# issues they surface are fixed (see docs/design/0002-synthetic-vision- +# fixtures.md). That is the point: they are a quality gate, and the work to +# make them pass is tracked on the integration-harness issue. Do not lower +# this floor to make the suite green — fix the model/prompt/pipeline until +# it clears the bar honestly. +ACCURACY_FLOOR = 1.0 + + +# --------------------------------------------------------------------------- +# Normalization for the scoring contract +# --------------------------------------------------------------------------- + + +def _norm(text: object) -> str: + """Case/whitespace-folded comparison key.""" + return re.sub(r"\s+", " ", str(text or "").strip()).casefold() + + +# Minimal position canonicalization — the model may phrase a few positions +# differently than the form's exact label. Keep this small and explicit. +_POSITION_CANON = { + "inspector of turns": "inspector of turns", + "turn inspector": "inspector of turns", + "turns inspector": "inspector of turns", + "inspector of turn": "inspector of turns", + "stroke judge": "stroke judge", + "stroke and turn judge": "stroke judge", + "chief timer": "chief timer", + "head timer": "chief timer", + "admin desk": "admin desk", + "administration desk": "admin desk", + "administrative desk": "admin desk", +} + + +def _norm_position(text: object) -> str: + key = _norm(text) + return _POSITION_CANON.get(key, key) + + +# --------------------------------------------------------------------------- +# Model-tag matching → tolerate ":latest" elision and bare-name forms +# --------------------------------------------------------------------------- + + +def _model_present(model: str, pulled: list[str]) -> bool: + """True if ``model`` matches any locally-pulled tag. + + Ollama elides ``:latest`` and accepts a bare name, so ``qwen2.5vl``, + ``qwen2.5vl:latest`` and ``qwen2.5vl:7b`` are all distinct strings we + must reconcile. We match if the requested tag, its bare name, or its + ``:latest`` form appears in the pulled set. + """ + base = model.split(":")[0] + wanted = {model, base, f"{base}:latest"} + return bool(set(pulled) & wanted) + + +# --------------------------------------------------------------------------- +# Run the vision pipeline once per module (the model call is slow) +# +# The fixture owns the daemon for the whole module: it enters OllamaDaemon +# (starting `ollama serve` only if one isn't already up), checks the model +# is pulled INSIDE the context, runs extraction+merge, then yields — so +# daemon teardown happens after every test in the module has run. +# --------------------------------------------------------------------------- + + +@pytest.fixture(scope="module") +def parsed(request): + """Real vision extraction + merge over the synthetic scan.""" + import ollama + + assert SCAN_PDF.is_file(), f"missing fixture {SCAN_PDF}" + + model = request.config.getoption("--vision-model") or vision_extract.DEFAULT_VISION_MODEL + allow_pull = request.config.getoption("--pull-models") + + # Binary on PATH? Skip cleanly on a box without Ollama installed. + try: + rt.find_binary() + except rt.OllamaBinaryMissingError as exc: + pytest.skip(f"Ollama not installed; {exc}") + + # Enter the daemon WITHOUT required_models so __enter__ never pulls or + # raises on a missing model — we want to control that ourselves below + # so a missing model skips (with a hint) instead of failing the suite. + try: + daemon = rt.OllamaDaemon(host=rt.DEFAULT_HOST).__enter__() + except rt.OllamaRuntimeError as exc: + pytest.skip(f"Ollama daemon unavailable: {exc}") + + try: + pulled = rt.list_pulled_models(host=rt.DEFAULT_HOST) + if not _model_present(model, pulled): + if allow_pull: + rt.ensure_models([model], host=rt.DEFAULT_HOST, auto_pull=True) + else: + pytest.skip( + f"Vision model {model!r} not pulled " + f"(have {sorted(pulled)}); run `ollama pull {model}` " + f"or rerun with `--pull-models` to download it." + ) + + template = TEMPLATES[TEMPLATE_ID] + client = daemon.client() + + pages = vision_extract.extract_pdf( + str(SCAN_PDF), + template, + client=client, + model=model, + use_cache=False, + ) + result = merge.merge( + pages, + source_pdf=SCAN_PDF.name, + template_id=TEMPLATE_ID, + template_confidence=1.0, + extraction_method="vision", + vision_model=model, + same_meet_checker=vision_extract.make_same_meet_checker(client, model), + ) + yield result + finally: + # Always tear the daemon down (stops it only if we started it). + daemon.__exit__(None, None, None) + + +@pytest.fixture(scope="module") +def golden(): + return json.loads(GOLDEN_PATH.read_text(encoding="utf-8")) + + +# --------------------------------------------------------------------------- +# Tests +# --------------------------------------------------------------------------- + + +def _index_by_name(evaluations): + out = {} + for ev in evaluations: + name = ev.official_name.value if ev.official_name else None + if name: + out[_norm(name)] = ev + return out + + +def test_finds_every_official(parsed, golden): + """Each golden official should appear in the extraction (matched by name).""" + by_name = _index_by_name(parsed.evaluations) + missing = [ + row["official_name"] + for row in golden["rows"] + if _norm(row["official_name"]) not in by_name + ] + # Allow a small miss on a clean scan, but most must be found. + found = len(golden["rows"]) - len(missing) + ratio = found / len(golden["rows"]) + assert ratio >= ACCURACY_FLOOR, ( + f"only matched {found}/{len(golden['rows'])} officials by name; " + f"missing: {missing}" + ) + + +def test_tuple_contract_accuracy(parsed, golden): + """(official, position, successful) tuples agree above the floor. + + Position is normalized; successful is compared exactly (True/False/None). + Reports every mismatch so a failure is actionable. + """ + by_name = _index_by_name(parsed.evaluations) + + total = 0 + correct = 0 + mismatches: list[str] = [] + + for row in golden["rows"]: + ev = by_name.get(_norm(row["official_name"])) + if ev is None: + # Unmatched official: counts against both position and successful. + total += 2 + mismatches.append(f"{row['official_name']!r}: not found in output") + continue + + # Position + total += 1 + got_pos = ev.position.value if ev.position else None + if _norm_position(got_pos) == _norm_position(row["position"]): + correct += 1 + else: + mismatches.append( + f"{row['official_name']!r} position: got {got_pos!r}, " + f"want {row['position']!r}" + ) + + # Successful (True / False / None) + total += 1 + got_succ = ev.successful.value if ev.successful else None + if got_succ == row["successful"]: + correct += 1 + else: + mismatches.append( + f"{row['official_name']!r} successful: got {got_succ!r}, " + f"want {row['successful']!r}" + ) + + accuracy = correct / total if total else 0.0 + assert accuracy >= ACCURACY_FLOOR, ( + f"tuple accuracy {accuracy:.2f} below floor {ACCURACY_FLOOR:.2f} " + f"({correct}/{total} correct).\nMismatches:\n " + + "\n ".join(mismatches) + ) + + +def test_meet_header(parsed, golden): + """Meet-level header should read back (case/space-insensitive).""" + g = golden["meet"] + got_name = parsed.meet.competition_name.value if parsed.meet.competition_name else None + got_host = parsed.meet.host_club.value if parsed.meet.host_club else None + # Host club is a short code that should match exactly; competition name + # is free text the model should at least contain the golden's words. + assert _norm(got_host) == _norm(g["host_club"]), ( + f"host_club: got {got_host!r}, want {g['host_club']!r}" + ) + assert _norm(g["competition_name"]) in _norm(got_name) or _norm(got_name) in _norm(g["competition_name"]), ( + f"competition_name: got {got_name!r}, want {g['competition_name']!r}" + )