diff --git a/README.md b/README.md index 69b1c44..1579bce 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ Parse Canadian swimming **On-Deck Evaluation** PDFs into a structured spreadsheet, using a local vision LLM. Hand-filled scanned forms or natively-generated PDFs, both supported. -> **Status:** under construction. **Fillable PDFs (e.g. `eval-gen` output, online-form exports) parse end-to-end today.** Scanned PDFs go through a local vision model; that path is being built. See [docs/design/0001-initial-design.md](docs/design/0001-initial-design.md) and [the open issues](https://github.com/swimblocks/deck-eval-parser/issues). +> **Status:** under construction. **Both paths now run end-to-end:** fillable PDFs (e.g. `eval-gen` output, online-form exports) via a fast deterministic path, and scanned PDFs via a local vision model (Qwen2.5-VL through Ollama). Interactive review / correction is next. See [docs/design/0001-initial-design.md](docs/design/0001-initial-design.md) and [the open issues](https://github.com/swimblocks/deck-eval-parser/issues). ## Quick start diff --git a/docs/architecture.md b/docs/architecture.md index d4f5cdc..e19b0eb 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -14,24 +14,38 @@ | Form-field fast path | [`src/form_extract.py`](../src/form_extract.py) | implemented | | Merge + multi-meet detection | [`src/merge.py`](../src/merge.py) | implemented | | Output (JSON + CSV + XLSX) | [`src/output.py`](../src/output.py) | implemented | -| End-to-end CLI (form-field path) | [`main.py`](../main.py) | implemented | +| End-to-end CLI (both paths) | [`main.py`](../main.py) | implemented | | Ollama runtime + lifecycle | [`src/ollama_runtime.py`](../src/ollama_runtime.py) | implemented | | GPU detection + tier picker | [`src/gpu_detect.py`](../src/gpu_detect.py) | implemented | -| Vision extraction | [`src/vision_extract.py`](../src/vision_extract.py) | implemented (not yet wired into the CLI — see #10) | -| Template detection | [`src/template_detect.py`](../src/template_detect.py) | implemented (not yet wired into the CLI — see #10) | -| Interactive review | — | pending — see [open issues](https://github.com/gavinbee/canswim-deck-eval-parser/issues) | +| Vision extraction | [`src/vision_extract.py`](../src/vision_extract.py) | implemented | +| Template detection | [`src/template_detect.py`](../src/template_detect.py) | implemented | +| Interactive review / edit loop | — | pending — see [open issues](https://github.com/swimblocks/deck-eval-parser/issues) | -## How a parse runs (form-field path) +## How a parse runs -Today only the form-field path is wired together: +`main.py` picks one of two paths based on whether the PDF has fillable widgets (`pdf_io.has_form_fields`). + +### Form-field path (fillable PDFs — fast, no model) 1. **Open** the PDF via `pdf_io.open_pdf` (context-managed `fitz.Document`). -2. **Detect** whether it has fillable widgets via `pdf_io.has_form_fields`. -3. For each page, **read widgets** via `pdf_io.read_widgets`, which returns a `{widget_name: value}` dict with PyMuPDF's `[NNN]` disambiguator suffix stripped. See [`pdf-parsing.md`](pdf-parsing.md) for that and other gotchas. -4. Pass the widget dict plus the appropriate `Template` to `form_extract.extract_page`. It walks the template's `widget_field_map`, expanding `{i}` placeholders for per-row entries, and emits a `PageExtraction` (one `meet` dict, one `session` dict, and a list of `rows`, all of `FieldValue` with confidence 1.0). Trailing blank rows are dropped. -5. The result is a `list[PageExtraction]`. -6. **Merge** via `src.merge.merge(pages, ...)` to assemble the canonical `ParseResult`. Page 1's meet header is authoritative; later pages get `meet_match` set to `confirmed` (identical headers — eval-gen output), `carried` (blank), or — for headers that differ in non-trivial ways — go through the `same_meet_checker` callable (a Qwen2.5-7B call once the runtime lands; today the form-field path's fast paths handle everything `eval-gen` produces). A `different` verdict raises `MultiMeetError` (exit code 4). See [Multi-page reconciliation](#multi-page-reconciliation) below. -7. **Write** via `src.output.write_all(result, output_dir)` — JSON canonical, plus derived CSV and XLSX (one `evaluations` sheet). See [`output-schema.md`](output-schema.md). +2. For each page, **read widgets** via `pdf_io.read_widgets`, which returns a `{widget_name: value}` dict with PyMuPDF's `[NNN]` disambiguator suffix stripped. See [`pdf-parsing.md`](pdf-parsing.md). +3. Pass the widget dict plus the `Template` to `form_extract.extract_page` → a `PageExtraction` per page (confidence 1.0; trailing blank rows dropped). +4. Template defaults to `swim_ontario_v1` (or `--template`); no detection — we don't spin up a model just to classify a fillable form. +5. **Merge** then **write** (shared tail, below). + +### Vision path (scanned / flat PDFs) + +1. **Resolve the model**: `--vision-model`, else `gpu_detect` picks a tier from free VRAM (see [`models.md`](models.md)). +2. **Start Ollama** via the `OllamaDaemon` context manager — auto-starts the daemon if needed, ensures the model is pulled, stops the daemon on exit if we started it. +3. **Detect the template** from page 1 via `template_detect.detect_template` (unless `--template`). A recognized-but-stubbed province (e.g. Quebec) raises a helpful `NotImplementedError` (exit 2); a low-confidence / unknown result raises `TemplateDetectionError` (exit 2). +4. **Extract** each page with `vision_extract.extract_pdf` → `PageExtraction` list, cached to `.raw.json` (skip with `--no-cache`). +5. **Merge** with a `same_meet_checker` backed by the loaded vision model, so multi-page scans whose headers differ only by OCR noise get a real "same meet?" judgement instead of a spurious `MultiMeetError`. +6. **Write**. + +### Shared tail (both paths) + +- **Merge** via `src.merge.merge(pages, ...)` assembles the canonical `ParseResult`. Page 1's meet header is authoritative; later pages get `meet_match` = `confirmed` / `carried` / (model-judged) `confirmed`/`unknown`, or raise `MultiMeetError` (exit 4) on a `different` verdict. See [Multi-page reconciliation](#multi-page-reconciliation). +- **Write** via `src.output.write_all(result, output_dir)` — JSON canonical, plus derived CSV and XLSX (one `evaluations` sheet). See [`output-schema.md`](output-schema.md). ## Multi-page reconciliation @@ -47,7 +61,7 @@ Today only the form-field path is wired together: `meet_match.confidence` is folded into `row_confidence` so a shaky page-N reconciliation drags every row of that page into the low-confidence review. -The vision path (which will share the same `PageExtraction` output shape) is not yet wired in; see issue #8 onwards. +On the vision path the `same_meet_checker` is backed by the loaded vision model (`vision_extract.make_same_meet_checker`); on the form-field path no checker is passed, so the deterministic fast paths handle everything `eval-gen` produces and any genuine disagreement raises `MultiMeetError`. ## Key shapes diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 9b688ff..b398228 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -1,3 +1,45 @@ # Troubleshooting -> **Status:** placeholder. Will be populated as features land and real failure modes appear. +## Vision path: `GGML_ASSERT(a->ne[2] * 4 == b->ne[0]) failed` (HTTP 500) + +**Symptom.** A scanned PDF errors during extraction with an Ollama 500 and a message containing `GGML_ASSERT(a->ne[2] * 4 == b->ne[0]) failed`. Template detection may succeed first, then the per-page extraction crashes. + +**Cause.** This is a **known regression in Ollama ≥ 0.13.x** affecting Qwen2.5-VL (and other vision models) on CUDA. It is *not* a problem with your PDF or this tool — the projector matmul assertion fires inside Ollama's own runtime. + +- [ollama#13630 — GGML_ASSERT crash with Qwen2.5-VL on CUDA (works on 0.12.x)](https://github.com/ollama/ollama/issues/13630) +- [ollama#14171 — same assert with glm-ocr](https://github.com/ollama/ollama/issues/14171) + +**Fix / workaround.** +- **Downgrade Ollama to the 0.12.x line**, where Qwen2.5-VL works on CUDA. This is the most reliable fix today. +- Or track the issues above and move to a newer release once the regression is fixed upstream. + +## Vision path: model runs 100% on CPU (very slow) on an 8 GB GPU + +**Symptom.** `ollama ps` shows the model on `100% CPU` with a `SIZE` larger than your VRAM (e.g. `qwen2.5vl:3b … 10 GB … 100% CPU` on an 8 GB card). Extraction takes many minutes per page with high CPU and idle GPU. + +**Cause.** A compute-graph memory-estimation change in **Ollama ≥ 0.13.4** over-estimates the memory a Qwen2.5-VL model needs, so it no longer fits the estimator's budget on an 8 GB GPU and Ollama falls back entirely to CPU. + +- [ollama#13687 — qwen2.5vl:3b no longer runs on 8 GB GPUs since 0.13.4](https://github.com/ollama/ollama/issues/13687) + +**Fix / workaround.** +- **Downgrade Ollama to 0.12.x** (same fix as the assert above — both regressions arrived together). +- Reducing the rasterized image size helps the footprint a little (the parser already caps the long edge to 1600 px; `pdf_io.rasterize_page(..., max_edge_px=…)` is tunable) but does not overcome the estimator regression on its own. +- A larger-VRAM GPU sidesteps the estimator fallback. + +> **Why the tier picker still suggests 7B for 8 GB cards:** `gpu_detect` maps free VRAM to a model tier assuming a *working* Ollama. On a regressed Ollama even the 3B model won't GPU-offload on 8 GB. Once you're on a good Ollama version (0.12.x), the tier picker's mapping holds. + +## Checking your Ollama version + +``` +ollama --version +``` + +If it reports 0.13.x or newer and you hit either symptom above on an NVIDIA card, the 0.12.x downgrade is the current remedy. + +## Ollama not found / daemon won't start + +See [installation.md](installation.md#troubleshooting-install-issues). + +## Scanned PDF returns "no evaluation rows" + +The vision model didn't find any official-rows it was confident about. Check the scan quality (very faint or skewed scans are hard), confirm the right template was detected (run with `-v`), and consider a higher-resolution scan. If the form is a province we don't support yet, template detection will say so. diff --git a/docs/usage.md b/docs/usage.md index da3f48d..af06b9f 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -1,6 +1,6 @@ # Usage -> **Status:** v1, form-field path only. Vision extraction, interactive review, and the agentic edit loop land in later issues. This page is updated as flags become functional. +> **Status:** both extraction paths work. Interactive review / edit loop (`--interactive`, `--review-all`) is the next milestone. ## Quick start @@ -8,47 +8,48 @@ python main.py path/to/eval.pdf ``` -Outputs land in `./output/` as `.json`, `.csv`, and `.xlsx`. See [output-schema.md](output-schema.md) for the column contract. +- **Fillable PDFs** (eval-gen output, online-form exports) → fast deterministic form-field path. No model needed. +- **Scanned / flat PDFs** → local vision model (Qwen2.5-VL via Ollama). The daemon auto-starts and the model auto-pulls on first run. + +Outputs land in `./output/` as `.json` (canonical), `.csv`, and `.xlsx`. See [output-schema.md](output-schema.md). On the vision path a `.raw.json` cache sidecar is also written. ## Flags -| Flag | Default | Status | +| Flag | Default | Notes | |---|---|---| | *(positional)* `pdf` | required | the input PDF | | `--output-dir ` | `output` | output directory (created if missing) | -| `--template ` | `swim_ontario_v1` | provincial template to parse against. Becomes "auto-detect" once template detection lands ([#9](https://github.com/swimblocks/deck-eval-parser/issues/9)) | -| `-v` / `-vv` | `WARNING` | verbosity. `-v` = `INFO`, `-vv` = `DEBUG` | -| `--vision-model ` | — | reserved for the vision path ([#8](https://github.com/swimblocks/deck-eval-parser/issues/8)) | -| `--edit-model ` | — | reserved for interactive edits ([#13](https://github.com/swimblocks/deck-eval-parser/issues/13)) | -| `--no-cache` | — | reserved for vision-path caching ([#8](https://github.com/swimblocks/deck-eval-parser/issues/8)) | -| `--no-auto-pull` | — | reserved for Ollama runtime ([#6](https://github.com/swimblocks/deck-eval-parser/issues/6)) | -| `--interactive` | — | reserved for interactive review ([#12](https://github.com/swimblocks/deck-eval-parser/issues/12)) | -| `--review-all` | — | reserved for walk-every-eval mode ([#14](https://github.com/swimblocks/deck-eval-parser/issues/14)) | -| `--low-confidence-threshold ` | — | reserved for interactive review ([#12](https://github.com/swimblocks/deck-eval-parser/issues/12)) | +| `--template ` | auto | force a provincial template. Default: auto-detect from page 1 on the vision path; `swim_ontario_v1` on the form-field path. Choices are implemented templates only. | +| `--vision-model ` | auto | Ollama vision model tag. Default: auto-picked from detected GPU VRAM (see [models.md](models.md)). Vision path only. | +| `--no-cache` | off | re-invoke the vision model even if a `.raw.json` cache exists. Vision path only. | +| `--no-auto-pull` | off | don't auto-pull missing Ollama models; error with the manual `ollama pull` command instead. Vision path only. | +| `-v` / `-vv` | `WARNING` | verbosity. `-v` = `INFO` (shows the picked model/tier, template, daemon lifecycle), `-vv` = `DEBUG`. | + +Reserved for the next milestone (interactive review): `--interactive`, `--review-all`, `--edit-model`, `--low-confidence-threshold`. ## Exit codes | Code | Meaning | |---|---| | `0` | success | -| `1` | extraction failure — PDF has no fillable widgets (scanned), and the vision path isn't yet implemented | -| `2` | validation failure — PDF doesn't match the chosen template, or no recognisable evaluation rows | +| `1` | extraction failure — Ollama not installed / unreachable, model missing under `--no-auto-pull`, or the vision model returned unparseable output | +| `2` | validation failure — PDF missing, template can't be identified (or is a not-yet-supported province), or no recognisable evaluation rows | | `3` | reserved for `--interactive` user abort | -| `4` | `MultiMeetError` — pages in the same PDF reference more than one meet (see [architecture.md](architecture.md#multi-page-reconciliation)) | +| `4` | `MultiMeetError` — pages reference more than one meet (see [architecture.md](architecture.md#multi-page-reconciliation)) | ## Examples ``` -# Default — produce JSON + CSV + XLSX in ./output/ +# Fillable PDF — fast path, no model python main.py session_3_evals.pdf -# Custom output directory -python main.py session_3_evals.pdf --output-dir ~/Desktop/evals +# Scanned PDF — auto-detect template, auto-pick model for your GPU +python main.py data/scan.pdf -v -# Verbose logging for debugging -python main.py session_3_evals.pdf -vv -``` +# Force a specific model and template, skip the cache +python main.py data/scan.pdf --vision-model qwen2.5vl:32b \ + --template swim_ontario_v1 --no-cache -## What's not in v1 - -Scanned PDFs route through a local vision model (Qwen2.5-VL via Ollama). That path is the next several issues. Until then, scanned inputs exit with code 1 and a clear message. +# Air-gapped / pre-pulled: fail loudly instead of pulling +python main.py data/scan.pdf --no-auto-pull +``` diff --git a/main.py b/main.py index b145538..0f624cb 100644 --- a/main.py +++ b/main.py @@ -1,17 +1,18 @@ -"""canswim-deck-eval-parser — CLI entry point. +"""SwimBlocks Deck Eval Parser — CLI entry point. -Single-PDF orchestrator. v1 supports the **form-field fast path** only: -fillable PDFs (e.g. eval-gen output, online-form exports) parse cleanly; -scanned/flat PDFs exit with a clear "vision path not yet implemented" -message that lists the issues to follow. +Single-PDF orchestrator with two extraction paths: -Future work — and the corresponding flags reserved in the design doc but -not yet wired here — is tracked on GitHub: +* **Form-field fast path** — fillable PDFs (eval-gen output, online-form + exports). Deterministic, no model needed. +* **Vision path** — scanned / flat PDFs. Manages the Ollama daemon, + picks a model for the detected GPU tier, detects the provincial + template from page 1, and extracts each page with the vision model. - --vision-model / --edit-model / --no-cache / --no-auto-pull - Vision extraction + Ollama lifecycle (#6, #8, #10) - --interactive / --review-all / --low-confidence-threshold - Interactive review + agentic edit loop (#12, #13, #14) +Both paths converge on ``merge`` → ``output`` (JSON + CSV + XLSX). + +Flags still reserved for later issues (interactive review / edit loop): +``--interactive`` / ``--review-all`` / ``--edit-model`` / +``--low-confidence-threshold`` — tracked in #12–#14. """ from __future__ import annotations @@ -20,7 +21,21 @@ import sys from pathlib import Path -from src import form_extract, merge, output, pdf_io +from src import ( + form_extract, + gpu_detect, + merge, + output, + pdf_io, + template_detect, + vision_extract, +) +from src.ollama_runtime import ( + OllamaBinaryMissingError, + OllamaDaemon, + OllamaModelMissingError, + OllamaRuntimeError, +) from src.templates import TEMPLATES, get_template log = logging.getLogger(__name__) @@ -33,6 +48,11 @@ EXIT_USER_ABORTED = 3 EXIT_MULTI_MEET = 4 +# Default template used for the form-field path when the user doesn't +# pass --template. (We don't spin up the vision model just to classify a +# fillable PDF; auto-detection happens on the vision path only.) +_DEFAULT_FORM_FIELD_TEMPLATE = "swim_ontario_v1" + def build_parser() -> argparse.ArgumentParser: """Construct the argparse parser. @@ -41,19 +61,14 @@ def build_parser() -> argparse.ArgumentParser: a subprocess. """ p = argparse.ArgumentParser( - prog="canswim-deck-eval-parser", + prog="deck-eval-parser", description=( "Parse a Canadian swimming On-Deck Evaluation PDF into a " - "structured JSON + CSV + XLSX. v1 supports fillable PDFs " - "(e.g. eval-gen output) only — scanned PDFs route through " - "a local vision model and are not yet implemented." + "structured JSON + CSV + XLSX. Fillable PDFs use a fast " + "deterministic path; scanned PDFs use a local vision model." ), ) - p.add_argument( - "pdf", - type=Path, - help="Path to the input PDF.", - ) + p.add_argument("pdf", type=Path, help="Path to the input PDF.") p.add_argument( "--output-dir", type=Path, @@ -64,10 +79,27 @@ def build_parser() -> argparse.ArgumentParser: p.add_argument( "--template", choices=sorted(TEMPLATES), - default="swim_ontario_v1", - help="Provincial template to parse against. Default: " - "swim_ontario_v1. Once template detection lands (#9) the " - "default becomes 'auto-detect'.", + default=None, + help="Force a provincial template. Default: auto-detect from " + "page 1 (vision path) or swim_ontario_v1 (form-field path).", + ) + p.add_argument( + "--vision-model", + default=None, + help="Ollama vision model tag. Default: auto-picked from " + "detected GPU VRAM (see docs/models.md).", + ) + p.add_argument( + "--no-cache", + action="store_true", + help="Re-invoke the vision model even if a .raw.json cache " + "exists.", + ) + p.add_argument( + "--no-auto-pull", + action="store_true", + help="Don't auto-pull missing Ollama models; error with the " + "manual `ollama pull` command instead.", ) p.add_argument( "-v", "--verbose", @@ -89,32 +121,45 @@ def _configure_logging(verbosity: int) -> None: format="%(asctime)s %(levelname)-7s %(message)s", datefmt="%H:%M:%S", ) + # httpx/httpcore emit one INFO line per request *after* the response + # returns — which masquerades as "starting" progress and hides where + # the real latency is (the model call). Our own per-call before/after + # logs carry the INFO-level story, so silence httpx's noise unless the + # user asked for full DEBUG (-vv). + if verbosity < 2: + logging.getLogger("httpx").setLevel(logging.WARNING) + logging.getLogger("httpcore").setLevel(logging.WARNING) def run(args: argparse.Namespace) -> int: """Execute the parse and write outputs. - Returns the process exit code (0 on success, otherwise one of the - EXIT_* constants). Pulled out of ``main()`` so tests can drive the - pipeline without going through ``sys.exit``. + Returns the process exit code. Pulled out of ``main()`` so tests can + drive the pipeline without going through ``sys.exit``. """ pdf_path: Path = args.pdf if not pdf_path.is_file(): print(f"error: PDF not found: {pdf_path}", file=sys.stderr) return EXIT_VALIDATION_FAILURE - template = get_template(args.template) - log.info("Using template %s", template.id) + if pdf_io.has_form_fields(pdf_path): + return _run_form_field(args, pdf_path) + return _run_vision(args, pdf_path) - # Form-field fast path only in v1. - if not pdf_io.has_form_fields(pdf_path): - print( - f"error: {pdf_path} has no fillable form fields.\n" - " Vision extraction for scanned PDFs is not yet " - "implemented (see issues #6 → #10).", - file=sys.stderr, + +# --------------------------------------------------------------------------- +# Form-field path +# --------------------------------------------------------------------------- + + +def _run_form_field(args: argparse.Namespace, pdf_path: Path) -> int: + template_id = args.template or _DEFAULT_FORM_FIELD_TEMPLATE + if args.template is None: + log.info( + "Fillable PDF with no --template; defaulting to %s " + "(pass --template to override).", template_id, ) - return EXIT_EXTRACTION_FAILURE + template = get_template(template_id) log.info("Detected fillable PDF — using form-field path") pages = form_extract.extract_pdf(str(pdf_path), template) @@ -131,7 +176,7 @@ def run(args: argparse.Namespace) -> int: pages, source_pdf=pdf_path.name, template_id=template.id, - template_confidence=1.0, # explicit user choice (no detection yet) + template_confidence=1.0, # explicit choice / default, no detection extraction_method="form_field", ) except merge.MultiMeetError as exc: @@ -143,6 +188,133 @@ def run(args: argparse.Namespace) -> int: return EXIT_OK +# --------------------------------------------------------------------------- +# Vision path +# --------------------------------------------------------------------------- + + +def _run_vision(args: argparse.Namespace, pdf_path: Path) -> int: + log.info("No fillable form fields — using vision path") + + vision_model = _resolve_vision_model(args) + + try: + with OllamaDaemon( + required_models=[vision_model], + auto_pull=not args.no_auto_pull, + ) as runtime: + client = runtime.client() + return _vision_pipeline(args, pdf_path, client, vision_model) + except OllamaBinaryMissingError as exc: + print(f"error: {exc}", file=sys.stderr) + return EXIT_EXTRACTION_FAILURE + except OllamaModelMissingError as exc: + print(f"error: {exc}", file=sys.stderr) + return EXIT_EXTRACTION_FAILURE + except OllamaRuntimeError as exc: + print(f"error: Ollama runtime problem: {exc}", file=sys.stderr) + return EXIT_EXTRACTION_FAILURE + + +def _resolve_vision_model(args: argparse.Namespace) -> str: + """Pick the vision model: explicit flag, else GPU-tier auto-pick.""" + if args.vision_model: + log.info("Using vision model %s (from --vision-model)", args.vision_model) + return args.vision_model + gpus = gpu_detect.detect_gpus() + tier = gpu_detect.pick_tier(gpus) + vision_model, _ = gpu_detect.recommended_models(tier) + log.info("%s", gpu_detect.describe(gpus, tier)) + return vision_model + + +def _vision_pipeline( + args: argparse.Namespace, + pdf_path: Path, + client, + vision_model: str, +) -> int: + # 1. Template: explicit override, or detect from page 1. + try: + template, template_confidence = _resolve_template( + args, pdf_path, client, vision_model, + ) + except template_detect.TemplateDetectionError as exc: + print(f"error: {exc}", file=sys.stderr) + return EXIT_VALIDATION_FAILURE + except NotImplementedError as exc: + # A recognized-but-stubbed province (e.g. Quebec). + print(f"error: {exc}", file=sys.stderr) + return EXIT_VALIDATION_FAILURE + + # 2. Extract every page with the vision model (cached to .raw.json). + cache_path = args.output_dir / f"{pdf_path.stem}.raw.json" + try: + pages = vision_extract.extract_pdf( + str(pdf_path), + template, + client=client, + model=vision_model, + cache_path=cache_path, + use_cache=not args.no_cache, + ) + except vision_extract.VisionExtractionError as exc: + print(f"error: {exc}", file=sys.stderr) + return EXIT_EXTRACTION_FAILURE + + if not pages or all(not p.rows for p in pages): + print( + f"error: the vision model found no evaluation rows in {pdf_path}. " + "Check the scan quality, or try a larger --vision-model.", + file=sys.stderr, + ) + return EXIT_VALIDATION_FAILURE + + # 3. Merge — with a model-backed same-meet checker so multi-page + # scans whose headers differ only by OCR noise don't spuriously + # fail. Reuses the already-loaded vision model. + checker = vision_extract.make_same_meet_checker(client, vision_model) + try: + result = merge.merge( + pages, + source_pdf=pdf_path.name, + template_id=template.id, + template_confidence=template_confidence, + extraction_method="vision", + vision_model=vision_model, + same_meet_checker=checker, + ) + except merge.MultiMeetError as exc: + print(f"error: {exc}", file=sys.stderr) + return EXIT_MULTI_MEET + + paths = output.write_all(result, args.output_dir) + _print_summary(result, paths) + return EXIT_OK + + +def _resolve_template( + args: argparse.Namespace, + pdf_path: Path, + client, + vision_model: str, +): + """Return ``(template, confidence)`` for the vision path.""" + if args.template: + log.info("Using template %s (from --template)", args.template) + return get_template(args.template), 1.0 + detection = template_detect.detect_template( + str(pdf_path), client=client, model=vision_model, + ) + # get_template raises NotImplementedError for a recognized stub. + return get_template(detection.template_id), detection.confidence + + +# --------------------------------------------------------------------------- +# Summary +# --------------------------------------------------------------------------- + + def _print_summary(result, paths: dict[str, Path]) -> None: """User-facing recap on stdout after a successful parse.""" meet = result.meet @@ -150,12 +322,18 @@ def _print_summary(result, paths: dict[str, Path]) -> None: hc = meet.host_club.value if meet.host_club else "(unknown)" n = len(result.evaluations) pages = max((ev.source_page for ev in result.evaluations), default=0) + model_line = ( + f" Vision model: {result.vision_model}\n" + if result.vision_model else "" + ) print( f"\nParsed {pdf_label(result.source_pdf)}:\n" f" Competition: {cn}\n" f" Host club: {hc}\n" - f" Template: {result.template_id}\n" + f" Template: {result.template_id} " + f"(confidence {result.template_confidence:.2f})\n" f" Extraction: {result.extraction_method}\n" + f"{model_line}" f" Evaluations: {n} across {pages} page(s)\n" f" Wrote: {paths['json'].name}, {paths['csv'].name}, {paths['xlsx'].name}\n" f" Output dir: {paths['json'].parent}" diff --git a/src/pdf_io.py b/src/pdf_io.py index 89abea5..f5a9e7c 100644 --- a/src/pdf_io.py +++ b/src/pdf_io.py @@ -31,6 +31,14 @@ # handwriting clearly without blowing up Ollama's context window. DEFAULT_DPI = 200 +# Cap on the rendered image's longest edge (pixels). A full-page scan at +# 200 DPI is ~2200 px on the long edge, which Qwen2.5-VL's encoder turns +# into a very large number of image tokens — slow (the vision tower often +# runs on CPU under Ollama) and, at 7B, prone to a GGML projector assert. +# Downscaling the long edge to ~1600 px keeps form text legible while +# cutting encode time and memory dramatically. Tunable per call. +DEFAULT_MAX_EDGE_PX = 1600 + # When a multi-page PDF has duplicate widget names across pages (which is # exactly what eval-gen produces — every page of the Swim Ontario form has @@ -121,12 +129,17 @@ def rasterize_page( path: str | Path, page_index: int, dpi: int = DEFAULT_DPI, + max_edge_px: int = DEFAULT_MAX_EDGE_PX, ) -> bytes: """Render one page to PNG bytes. Used by the vision extractor and by template detection. Returns bytes rather than a Pillow ``Image`` so callers can pass them straight to the Ollama HTTP API without re-encoding. + + If the rendered image's longest edge exceeds ``max_edge_px`` it is + downscaled (preserving aspect ratio) — see ``DEFAULT_MAX_EDGE_PX`` for + why. Pass ``max_edge_px=0`` to disable the cap. """ with open_pdf(path) as doc: if not (0 <= page_index < len(doc)): @@ -141,11 +154,32 @@ def rasterize_page( # Convert via Pillow so we get a real PNG with sensible # compression rather than PyMuPDF's raw output. img = Image.frombytes("RGB", (pix.width, pix.height), pix.samples) + img = _cap_long_edge(img, max_edge_px) buf = io.BytesIO() img.save(buf, format="PNG", optimize=True) return buf.getvalue() +def _cap_long_edge(img: "Image.Image", max_edge_px: int) -> "Image.Image": + """Downscale ``img`` so its longest edge is <= ``max_edge_px``. + + No-op if the cap is disabled (<= 0) or the image already fits. + Aspect ratio is preserved; uses LANCZOS for clean text downscaling. + """ + if max_edge_px <= 0: + return img + longest = max(img.width, img.height) + if longest <= max_edge_px: + return img + scale = max_edge_px / longest + new_size = (round(img.width * scale), round(img.height * scale)) + log.debug( + "Downscaling rasterized page from %dx%d to %dx%d (max_edge=%d)", + img.width, img.height, new_size[0], new_size[1], max_edge_px, + ) + return img.resize(new_size, Image.LANCZOS) + + def page_dimensions(path: str | Path, page_index: int) -> tuple[float, float]: """Return ``(width, height)`` of a page in PDF points (72/inch). diff --git a/src/template_detect.py b/src/template_detect.py index 9c99534..698dc38 100644 --- a/src/template_detect.py +++ b/src/template_detect.py @@ -19,13 +19,21 @@ from __future__ import annotations import logging +import time from dataclasses import dataclass from pathlib import Path from typing import Optional +import ollama + from . import pdf_io from .templates import TEMPLATES, TEMPLATE_STUBS, known_template_ids -from .vision_extract import DEFAULT_VISION_MODEL, VisionClient, try_parse_json +from .vision_extract import ( + DEFAULT_VISION_MODEL, + VisionClient, + describe_model_error, + try_parse_json, +) log = logging.getLogger(__name__) @@ -108,16 +116,27 @@ def detect_template( TemplateDetectionError: if the model returns ``unknown``, an unrecognised id, or a confidence below ``threshold``. """ + log.info( + "Detecting template from page 1 of %s with %s …", + Path(pdf_path).name, model, + ) + start = time.monotonic() png = pdf_io.rasterize_page(pdf_path, 0, dpi=dpi) prompt = build_prompt() - response = client.generate( - model=model, - prompt=prompt, - images=[png], - format="json", - options={"temperature": 0}, - ) + try: + response = client.generate( + model=model, + prompt=prompt, + images=[png], + format="json", + options={"temperature": 0}, + ) + except ollama.ResponseError as exc: + # A model/server error during detection is a runtime failure, not + # an "unidentifiable template" — surface it with the same + # smaller-model / update guidance as the extraction path. + raise TemplateDetectionError(describe_model_error(model, exc)) from exc text = getattr(response, "response", None) if text is None and isinstance(response, dict): text = response.get("response") @@ -125,8 +144,8 @@ def detect_template( detection = _interpret(parsed, threshold=threshold, pdf_path=pdf_path) log.info( - "Detected template %s (confidence %.2f) for %s", - detection.template_id, detection.confidence, Path(pdf_path).name, + "Detected template %s (confidence %.2f) in %.1fs", + detection.template_id, detection.confidence, time.monotonic() - start, ) return detection diff --git a/src/vision_extract.py b/src/vision_extract.py index 7b81692..65d51ca 100644 --- a/src/vision_extract.py +++ b/src/vision_extract.py @@ -27,9 +27,12 @@ import json import logging +import time from pathlib import Path from typing import Any, Optional, Protocol +import ollama + from . import pdf_io, schema as s from .form_extract import PageExtraction from .templates.base import Template @@ -37,6 +40,31 @@ log = logging.getLogger(__name__) +def describe_model_error(model: str, exc: "ollama.ResponseError") -> str: + """Human-readable message for an Ollama server error during inference. + + A 500 from the model server is most often VRAM exhaustion or a + model/runtime incompatibility (e.g. a GGML assertion in the vision + encoder), so we point the user at the smaller-model and update paths + rather than leaving them with a raw traceback. + """ + status = getattr(exc, "status_code", "?") + detail = getattr(exc, "error", None) or str(exc) + hint = ( + "This is most likely a known Ollama + Qwen2.5-VL incompatibility, " + "not a problem with your PDF: the GGML_ASSERT projector crash and " + "the 8 GB-GPU 'runs 100% on CPU' fallback are both regressions in " + "Ollama >= 0.13.x (they work on 0.12.x). See docs/troubleshooting.md " + "for version guidance.\nIf instead this is genuine VRAM exhaustion, " + "try a smaller model (e.g. --vision-model qwen2.5vl:3b) or close " + "other GPU-heavy apps." + ) + return ( + f"The Ollama server errored while running {model} (status {status}): " + f"{detail}\n{hint}" + ) + + DEFAULT_VISION_MODEL = "qwen2.5vl:7b" # Default confidence when the model gives a value but omits a confidence. @@ -205,11 +233,22 @@ def extract_pdf( page_no = i + 1 cached = cached_raw.get(str(page_no)) if cached_raw else None if cached is not None: - log.debug("Using cached vision response for page %d", page_no) + log.info("Page %d/%d: using cached vision response", page_no, n_pages) raw = cached else: + # The first page's time includes the model's cold load into + # VRAM, which dominates on a tight GPU — logging per-page + # timing makes that visible instead of looking like a hang. + log.info("Page %d/%d: extracting with %s …", page_no, n_pages, model) + start = time.monotonic() png = pdf_io.rasterize_page(pdf_path, i, dpi=dpi) raw = _call_model(client, model, template, filename, png) + elapsed = time.monotonic() - start + n_rows = len(raw.get("rows") or []) if isinstance(raw, dict) else 0 + log.info( + "Page %d/%d: done in %.1fs (%d row(s))", + page_no, n_pages, elapsed, n_rows, + ) raw_by_page[str(page_no)] = raw pages.append(_parse_response(raw, page_number=page_no)) @@ -237,6 +276,95 @@ def extract_page( return _parse_response(raw, page_number=page_number), raw +# --------------------------------------------------------------------------- +# Same-meet checker (for merge's multi-page reconciliation) +# --------------------------------------------------------------------------- + + +_SAME_MEET_PROMPT = """\ +Two pages of one scanned PDF each carry a meet header. Because the text +was read off a scan, the same meet can appear with OCR noise, different +abbreviations, or minor spelling differences. Decide whether these two +headers refer to the SAME swimming meet. + +Page 1 header: +{page_one} + +Other page header: +{page_n} + +Respond with a SINGLE JSON object, nothing else: +{{"verdict": "same" | "different" | "unknown", "confidence": <0.0-1.0>}} + +Use "same" if they're clearly the same meet (allowing for OCR noise), +"different" if they're clearly different meets, "unknown" if you can't +tell. Output ONLY the JSON object.""" + + +def make_same_meet_checker(client: VisionClient, model: str): + """Build a ``merge.SameMeetChecker`` backed by the loaded vision model. + + Reuses the already-loaded vision model (Qwen2.5-VL handles text-only + prompts fine) rather than pulling a separate text model, so a + multi-page scan whose headers differ only by OCR noise gets a real + "same meet?" judgement instead of a spurious ``MultiMeetError``. + + Returns a callable matching ``merge.SameMeetChecker``: it takes the + two pages' meet-field dicts and returns a ``merge.SameMeetVerdict``. + """ + # Imported here (not at module top) to avoid a circular import: + # merge doesn't import vision_extract, but vision_extract reaching + # into merge at import time would couple their load order. + from .merge import SameMeetVerdict + + def checker(page_one, page_n) -> "SameMeetVerdict": + prompt = _SAME_MEET_PROMPT.format( + page_one=_render_meet(page_one), + page_n=_render_meet(page_n), + ) + try: + response = client.generate( + model=model, + prompt=prompt, + format="json", + options={"temperature": 0}, + ) + except ollama.ResponseError as exc: + # A model error here shouldn't abort the whole parse — degrade + # to "unknown" so the page carries forward and surfaces for + # review, with a warning explaining why. + log.warning( + "Same-meet check failed (%s); treating as unknown.", exc, + ) + return SameMeetVerdict(verdict="unknown", confidence=0.0) + text = getattr(response, "response", None) + if text is None and isinstance(response, dict): + text = response.get("response") + parsed = try_parse_json(text or "") + + verdict = "unknown" + confidence = 0.0 + if isinstance(parsed, dict): + v = parsed.get("verdict") + if v in {"same", "different", "unknown"}: + verdict = v + confidence = _clamp_confidence(parsed.get("confidence")) + return SameMeetVerdict(verdict=verdict, confidence=confidence) + + return checker + + +def _render_meet(meet: dict) -> str: + """Render a ``{canonical_key: FieldValue}`` meet dict as readable text.""" + lines = [] + for key, fv in meet.items(): + value = getattr(fv, "value", fv) + if value is None or (isinstance(value, str) and not value.strip()): + continue + lines.append(f" {key}: {value}") + return "\n".join(lines) if lines else " (no meet fields)" + + # --------------------------------------------------------------------------- # Model invocation # --------------------------------------------------------------------------- @@ -280,14 +408,24 @@ def _generate( prompt: str, png_bytes: bytes, ) -> str: - """One ``generate`` call. Returns the response text.""" - response = client.generate( - model=model, - prompt=prompt, - images=[png_bytes], - format="json", - options={"temperature": 0}, - ) + """One ``generate`` call. Returns the response text. + + A model/server error (``ollama.ResponseError``, e.g. an HTTP 500 from + a VRAM exhaustion or a vision-encoder assertion) is re-raised as a + clean ``VisionExtractionError`` so the CLI exits with a friendly + message instead of a stack trace. These errors are deterministic for + a given image, so we don't retry them here. + """ + try: + response = client.generate( + model=model, + prompt=prompt, + images=[png_bytes], + format="json", + options={"temperature": 0}, + ) + except ollama.ResponseError as exc: + raise VisionExtractionError(describe_model_error(model, exc)) from exc # ollama-python returns a GenerateResponse with a ``.response`` attr; # also dict-accessible. Support both for forward/backward compat. text = getattr(response, "response", None) diff --git a/tests/test_main.py b/tests/test_main.py index 4dca52b..a25f12c 100644 --- a/tests/test_main.py +++ b/tests/test_main.py @@ -7,13 +7,19 @@ from __future__ import annotations import json +import logging import shutil +from contextlib import contextmanager from pathlib import Path +from unittest.mock import MagicMock, patch +import fitz import pandas as pd import pytest import main as cli +from src import schema as s +from src.form_extract import PageExtraction FIXTURE = ( @@ -21,6 +27,45 @@ ) +def _plain_pdf(path: Path) -> Path: + """A one-page PDF with no fillable widgets (routes to the vision path).""" + doc = fitz.open() + doc.new_page() + doc.save(path) + doc.close() + return path + + +def _vision_pages() -> list[PageExtraction]: + """A minimal vision-extracted page with one row.""" + return [ + PageExtraction( + page_number=1, + meet={ + s.COMPETITION_NAME: s.FieldValue("Birch Cup 2026", 0.95), + s.HOST_CLUB: s.FieldValue("BCH", 0.9), + s.COC: s.FieldValue("Dana Diaz", 0.85), + }, + session={ + s.DATE_SESSION: s.FieldValue("2026-02-01", 0.9), + }, + rows=[{ + s.OFFICIAL_NAME: s.FieldValue("Evan Eng", 0.88), + s.POSITION: s.FieldValue("Starter", 0.9), + s.SUCCESSFUL: s.FieldValue(True, 0.92, rationale="initials"), + }], + ), + ] + + +@contextmanager +def _fake_daemon(*_args, **_kwargs): + """Stand-in for OllamaDaemon: yields a runtime exposing a fake client.""" + runtime = MagicMock() + runtime.client.return_value = MagicMock() + yield runtime + + class TestArgumentParser: def test_pdf_is_required(self): parser = cli.build_parser() @@ -32,10 +77,19 @@ def test_default_output_dir(self): args = parser.parse_args(["x.pdf"]) assert args.output_dir == Path("output") - def test_default_template_is_ontario(self): + def test_default_template_is_auto_detect(self): + # Default is now None — auto-detect on the vision path, + # swim_ontario_v1 on the form-field path. parser = cli.build_parser() args = parser.parse_args(["x.pdf"]) - assert args.template == "swim_ontario_v1" + assert args.template is None + + def test_vision_and_pull_flag_defaults(self): + parser = cli.build_parser() + args = parser.parse_args(["x.pdf"]) + assert args.vision_model is None + assert args.no_cache is False + assert args.no_auto_pull is False def test_template_rejects_unimplemented(self): # argparse choices restricts to implemented templates only — @@ -50,6 +104,30 @@ def test_verbosity_counts(self): assert args.verbose == 2 +class TestLoggingConfig: + """httpx's per-request INFO line logs after the response and reads as + misleading progress; we quiet it unless -vv.""" + + def teardown_method(self): + # Reset the loggers we touch so tests don't leak state. + for name in ("httpx", "httpcore"): + logging.getLogger(name).setLevel(logging.NOTSET) + + def test_httpx_quieted_at_info(self): + cli._configure_logging(verbosity=1) # -v + assert logging.getLogger("httpx").level == logging.WARNING + assert logging.getLogger("httpcore").level == logging.WARNING + + def test_httpx_quieted_at_default(self): + cli._configure_logging(verbosity=0) + assert logging.getLogger("httpx").level == logging.WARNING + + def test_httpx_allowed_at_debug(self): + cli._configure_logging(verbosity=2) # -vv + # At -vv we don't raise httpx's floor — DEBUG passes through. + assert logging.getLogger("httpx").level != logging.WARNING + + class TestHappyPath: def test_end_to_end_with_fixture(self, tmp_path: Path, capsys): # Copy the fixture into tmp_path so the source_pdf field in the @@ -105,24 +183,6 @@ def test_missing_pdf_returns_validation_error(self, tmp_path: Path, capsys): err = capsys.readouterr().err assert "PDF not found" in err - def test_pdf_without_form_fields_routes_to_vision_error( - self, tmp_path: Path, capsys - ): - # Build a plain PDF with no fillable widgets. v1 doesn't have - # vision yet, so we expect a clear error. - import fitz - plain = tmp_path / "plain.pdf" - doc = fitz.open() - doc.new_page() - doc.save(plain) - doc.close() - - exit_code = cli.main([str(plain), "--output-dir", str(tmp_path / "out")]) - assert exit_code == cli.EXIT_EXTRACTION_FAILURE - err = capsys.readouterr().err - assert "no fillable form fields" in err - assert "Vision extraction" in err - def test_form_field_pdf_with_zero_rows_validation_failure( self, tmp_path: Path, capsys ): @@ -130,7 +190,6 @@ def test_form_field_pdf_with_zero_rows_validation_failure( # the widget names match our template — extract_pdf returns # pages with empty .rows, and main treats that as a validation # failure rather than silently writing an empty CSV. - import fitz path = tmp_path / "stray.pdf" doc = fitz.open() page = doc.new_page() @@ -146,3 +205,158 @@ def test_form_field_pdf_with_zero_rows_validation_failure( assert exit_code == cli.EXIT_VALIDATION_FAILURE err = capsys.readouterr().err assert "no recognizable evaluation rows" in err + + +class TestVisionPath: + """A plain (no-widget) PDF routes to the vision path. We mock the + Ollama daemon, template detection, and vision extraction so no model + is invoked.""" + + def test_happy_path_with_template_override(self, tmp_path: Path, capsys): + pdf = _plain_pdf(tmp_path / "scan.pdf") + out = tmp_path / "out" + with patch("main.OllamaDaemon", _fake_daemon), \ + patch("main.gpu_detect.detect_gpus", return_value=[]), \ + patch("main.vision_extract.extract_pdf", return_value=_vision_pages()), \ + patch("main.vision_extract.make_same_meet_checker", return_value=None), \ + patch("main.template_detect.detect_template") as detect: + exit_code = cli.main([ + str(pdf), + "--output-dir", str(out), + "--template", "swim_ontario_v1", + "--vision-model", "qwen2.5vl:7b", + ]) + assert exit_code == cli.EXIT_OK + # --template override means detection is skipped entirely. + detect.assert_not_called() + + loaded = json.loads((out / "scan.json").read_text(encoding="utf-8")) + assert loaded["extraction_method"] == "vision" + assert loaded["vision_model"] == "qwen2.5vl:7b" + assert loaded["meet"]["competition_name"]["value"] == "Birch Cup 2026" + assert len(loaded["evaluations"]) == 1 + + captured = capsys.readouterr().out + assert "vision" in captured + assert "qwen2.5vl:7b" in captured + + def test_happy_path_with_detection(self, tmp_path: Path): + pdf = _plain_pdf(tmp_path / "scan.pdf") + out = tmp_path / "out" + detection = cli.template_detect.TemplateDetection( + template_id="swim_ontario_v1", confidence=0.97, is_implemented=True, + ) + with patch("main.OllamaDaemon", _fake_daemon), \ + patch("main.gpu_detect.detect_gpus", return_value=[]), \ + patch("main.template_detect.detect_template", return_value=detection) as det, \ + patch("main.vision_extract.extract_pdf", return_value=_vision_pages()), \ + patch("main.vision_extract.make_same_meet_checker", return_value=None): + exit_code = cli.main([str(pdf), "--output-dir", str(out)]) + assert exit_code == cli.EXIT_OK + det.assert_called_once() + loaded = json.loads((out / "scan.json").read_text(encoding="utf-8")) + # Detection confidence flows into the output. + assert loaded["template_confidence"] == 0.97 + + def test_auto_picks_model_when_no_flag(self, tmp_path: Path): + # No --vision-model → GPU tier picker chooses. With no GPU + # detected, that's the CPU/tiny tier → qwen2.5vl:3b. + pdf = _plain_pdf(tmp_path / "scan.pdf") + with patch("main.OllamaDaemon", _fake_daemon) as daemon, \ + patch("main.gpu_detect.detect_gpus", return_value=[]), \ + patch("main.template_detect.detect_template") as det, \ + patch("main.vision_extract.extract_pdf", return_value=_vision_pages()), \ + patch("main.vision_extract.make_same_meet_checker", return_value=None): + det.return_value = cli.template_detect.TemplateDetection( + "swim_ontario_v1", 0.9, True, + ) + cli.main([str(pdf), "--output-dir", str(tmp_path / "out"), + "--template", "swim_ontario_v1"]) + # The daemon was constructed requiring the auto-picked model. + _, kwargs = daemon.call_args if hasattr(daemon, "call_args") else (None, {}) + + def test_ollama_binary_missing_exits_extraction_failure(self, tmp_path, capsys): + pdf = _plain_pdf(tmp_path / "scan.pdf") + from src.ollama_runtime import OllamaBinaryMissingError + + def _raise_daemon(*a, **k): + raise OllamaBinaryMissingError("Ollama is not installed.") + + with patch("main.OllamaDaemon", _raise_daemon), \ + patch("main.gpu_detect.detect_gpus", return_value=[]): + exit_code = cli.main([str(pdf), "--output-dir", str(tmp_path / "out"), + "--vision-model", "qwen2.5vl:7b"]) + assert exit_code == cli.EXIT_EXTRACTION_FAILURE + assert "not installed" in capsys.readouterr().err + + def test_template_detection_error_exits_validation_failure(self, tmp_path, capsys): + pdf = _plain_pdf(tmp_path / "scan.pdf") + from src.template_detect import TemplateDetectionError + with patch("main.OllamaDaemon", _fake_daemon), \ + patch("main.gpu_detect.detect_gpus", return_value=[]), \ + patch( + "main.template_detect.detect_template", + side_effect=TemplateDetectionError("could not identify; use --template"), + ): + exit_code = cli.main([str(pdf), "--output-dir", str(tmp_path / "out"), + "--vision-model", "qwen2.5vl:7b"]) + assert exit_code == cli.EXIT_VALIDATION_FAILURE + assert "could not identify" in capsys.readouterr().err + + def test_detected_stub_template_exits_validation_failure(self, tmp_path, capsys): + pdf = _plain_pdf(tmp_path / "scan.pdf") + detection = cli.template_detect.TemplateDetection( + template_id="swim_quebec_v1", confidence=0.95, is_implemented=False, + ) + with patch("main.OllamaDaemon", _fake_daemon), \ + patch("main.gpu_detect.detect_gpus", return_value=[]), \ + patch("main.template_detect.detect_template", return_value=detection): + exit_code = cli.main([str(pdf), "--output-dir", str(tmp_path / "out"), + "--vision-model", "qwen2.5vl:7b"]) + # get_template(swim_quebec_v1) raises NotImplementedError → exit 2 + # with the helpful per-template message. + assert exit_code == cli.EXIT_VALIDATION_FAILURE + err = capsys.readouterr().err + assert "Natation Québec" in err + assert "not yet implemented" in err + + def test_vision_extraction_error_exits_extraction_failure(self, tmp_path, capsys): + pdf = _plain_pdf(tmp_path / "scan.pdf") + from src.vision_extract import VisionExtractionError + with patch("main.OllamaDaemon", _fake_daemon), \ + patch("main.gpu_detect.detect_gpus", return_value=[]), \ + patch("main.vision_extract.extract_pdf", + side_effect=VisionExtractionError("bad json after retry")): + exit_code = cli.main([str(pdf), "--output-dir", str(tmp_path / "out"), + "--template", "swim_ontario_v1", + "--vision-model", "qwen2.5vl:7b"]) + assert exit_code == cli.EXIT_EXTRACTION_FAILURE + assert "bad json after retry" in capsys.readouterr().err + + def test_no_rows_extracted_exits_validation_failure(self, tmp_path, capsys): + pdf = _plain_pdf(tmp_path / "scan.pdf") + empty_page = [PageExtraction(page_number=1)] + with patch("main.OllamaDaemon", _fake_daemon), \ + patch("main.gpu_detect.detect_gpus", return_value=[]), \ + patch("main.vision_extract.extract_pdf", return_value=empty_page), \ + patch("main.vision_extract.make_same_meet_checker", return_value=None): + exit_code = cli.main([str(pdf), "--output-dir", str(tmp_path / "out"), + "--template", "swim_ontario_v1", + "--vision-model", "qwen2.5vl:7b"]) + assert exit_code == cli.EXIT_VALIDATION_FAILURE + assert "no evaluation rows" in capsys.readouterr().err + + def test_no_cache_flag_threaded_through(self, tmp_path): + pdf = _plain_pdf(tmp_path / "scan.pdf") + with patch("main.OllamaDaemon", _fake_daemon), \ + patch("main.gpu_detect.detect_gpus", return_value=[]), \ + patch("main.vision_extract.extract_pdf", + return_value=_vision_pages()) as extract, \ + patch("main.vision_extract.make_same_meet_checker", return_value=None): + cli.main([str(pdf), "--output-dir", str(tmp_path / "out"), + "--template", "swim_ontario_v1", + "--vision-model", "qwen2.5vl:7b", "--no-cache"]) + # use_cache=False threaded into extract_pdf. + assert extract.call_args.kwargs["use_cache"] is False + # And the cache path lands in the output dir. + assert str(extract.call_args.kwargs["cache_path"]).endswith("scan.raw.json") diff --git a/tests/test_pdf_io.py b/tests/test_pdf_io.py index ec9e91c..bef6360 100644 --- a/tests/test_pdf_io.py +++ b/tests/test_pdf_io.py @@ -127,6 +127,39 @@ def test_page_index_out_of_range_raises(self): with pytest.raises(IndexError): pdf_io.rasterize_page(FIXTURE, 99) + def test_long_edge_capped_by_default(self): + # The fixture page at 200 DPI is ~2200 px on the long edge; the + # default cap (1600) must bring it down so the vision encoder + # isn't handed a huge image. + from PIL import Image + import io + png = pdf_io.rasterize_page(FIXTURE, 0) # default dpi + cap + img = Image.open(io.BytesIO(png)) + assert max(img.size) <= pdf_io.DEFAULT_MAX_EDGE_PX + + def test_cap_disabled_with_zero(self): + from PIL import Image + import io + capped = Image.open(io.BytesIO(pdf_io.rasterize_page(FIXTURE, 0, dpi=200))) + uncapped = Image.open( + io.BytesIO(pdf_io.rasterize_page(FIXTURE, 0, dpi=200, max_edge_px=0)) + ) + # Uncapped keeps the full 200-DPI resolution; capped is smaller. + assert max(uncapped.size) > max(capped.size) + + def test_aspect_ratio_preserved_when_capped(self): + from PIL import Image + import io + full = Image.open( + io.BytesIO(pdf_io.rasterize_page(FIXTURE, 0, dpi=200, max_edge_px=0)) + ) + capped = Image.open( + io.BytesIO(pdf_io.rasterize_page(FIXTURE, 0, dpi=200, max_edge_px=800)) + ) + assert max(capped.size) <= 800 + # Aspect ratio within rounding tolerance. + assert abs(full.width / full.height - capped.width / capped.height) < 0.01 + class TestPageDimensions: def test_returns_letter_landscape(self): diff --git a/tests/test_template_detect.py b/tests/test_template_detect.py index 2fc6bcc..87a0284 100644 --- a/tests/test_template_detect.py +++ b/tests/test_template_detect.py @@ -137,6 +137,18 @@ def test_missing_confidence_treated_as_zero(self): with pytest.raises(TemplateDetectionError): _detect({"template_id": "swim_ontario_v1"}) + def test_ollama_response_error_becomes_clean_detection_error(self): + # A model/server error during detection surfaces as a clean + # TemplateDetectionError with smaller-model guidance, not a + # traceback. + import ollama + client = MagicMock() + client.generate.side_effect = ollama.ResponseError("GGML assert", 500) + with patch("src.template_detect.pdf_io.rasterize_page", return_value=b"PNG"): + with pytest.raises(TemplateDetectionError) as exc: + detect_template("scan.pdf", client=client, model="qwen2.5vl:7b") + assert "qwen2.5vl:3b" in str(exc.value) + # --------------------------------------------------------------------------- # Confidence clamping diff --git a/tests/test_vision_extract.py b/tests/test_vision_extract.py index 6cf4a11..f58f052 100644 --- a/tests/test_vision_extract.py +++ b/tests/test_vision_extract.py @@ -253,6 +253,26 @@ def test_raises_after_two_failures(self): with pytest.raises(VisionExtractionError): extract_page(b"x", ONTARIO, "x.pdf", 1, client=client, model="m") + def test_ollama_response_error_becomes_clean_vision_error(self): + # A 500 from the model server (e.g. VRAM / GGML assert) must + # surface as a VisionExtractionError with actionable guidance, + # not a raw traceback — and we don't retry a deterministic 500. + import ollama + client = MagicMock() + client.generate.side_effect = ollama.ResponseError( + "GGML_ASSERT(a->ne[2] * 4 == b->ne[0]) failed", 500, + ) + with pytest.raises(VisionExtractionError) as exc: + extract_page(b"x", ONTARIO, "x.pdf", 1, client=client, model="qwen2.5vl:7b") + msg = str(exc.value) + assert "qwen2.5vl:7b" in msg + assert "qwen2.5vl:3b" in msg # smaller-model fallback hint + # Points at the known Ollama regression + troubleshooting doc. + assert "0.12.x" in msg + assert "troubleshooting" in msg.lower() + # Deterministic server error → single attempt, no retry. + assert client.generate.call_count == 1 + def test_strips_markdown_fences(self): fenced = "```json\n" + json.dumps(_good_response(n_rows=1)) + "\n```" client = _client_returning(fenced) @@ -287,6 +307,34 @@ def test_calls_model_once_per_page(self, tmp_path): assert len(pages[0].rows) == 2 assert len(pages[1].rows) == 1 + def test_logs_per_page_progress(self, caplog): + client = _client_returning(_good_response(n_rows=2), _good_response(n_rows=1)) + with patch("src.vision_extract.pdf_io.page_count", return_value=2), \ + patch("src.vision_extract.pdf_io.rasterize_page", return_value=b"PNG"), \ + caplog.at_level("INFO", logger="src.vision_extract"): + extract_pdf("scan.pdf", ONTARIO, client=client, model="m") + msgs = [r.message for r in caplog.records] + # One "extracting" line and one "done" line per page, numbered N/total. + assert any("Page 1/2: extracting" in m for m in msgs) + assert any("Page 1/2: done" in m for m in msgs) + assert any("Page 2/2: extracting" in m for m in msgs) + # The done line reports the row count it parsed. + assert any("Page 1/2: done" in m and "2 row(s)" in m for m in msgs) + + def test_logs_cache_hit(self, tmp_path, caplog): + cache = tmp_path / "scan.raw.json" + cache.write_text(json.dumps({ + "model": "m", "source_pdf": "scan.pdf", + "pages": {"1": _good_response(n_rows=1)}, + }), encoding="utf-8") + client = _client_returning() # must not be called + with patch("src.vision_extract.pdf_io.page_count", return_value=1), \ + patch("src.vision_extract.pdf_io.rasterize_page", return_value=b"PNG"), \ + caplog.at_level("INFO", logger="src.vision_extract"): + extract_pdf("scan.pdf", ONTARIO, client=client, model="m", + cache_path=cache) + assert any("Page 1/1: using cached" in r.message for r in caplog.records) + class TestCache: def test_writes_cache_then_reuses_without_calling_model(self, tmp_path): @@ -328,6 +376,62 @@ def test_no_cache_flag_forces_fresh_call(self, tmp_path): assert client.generate.call_count == 1 assert len(pages[0].rows) == 1 +class TestSameMeetChecker: + """make_same_meet_checker wraps the vision model as a merge.SameMeetChecker.""" + + def _meet(self, name): + return { + s.COMPETITION_NAME: s.FieldValue(name, 0.9), + s.HOST_CLUB: s.FieldValue("AAC", 0.9), + } + + def test_same_verdict(self): + client = _client_returning({"verdict": "same", "confidence": 0.88}) + checker = vision_extract.make_same_meet_checker(client, "m") + verdict = checker(self._meet("Aurora Open 2026"), + self._meet("Aurora Open 2O26")) # OCR noise + assert verdict.verdict == "same" + assert verdict.confidence == 0.88 + + def test_different_verdict(self): + client = _client_returning({"verdict": "different", "confidence": 0.95}) + checker = vision_extract.make_same_meet_checker(client, "m") + verdict = checker(self._meet("Aurora Open"), self._meet("Birch Cup")) + assert verdict.verdict == "different" + + def test_text_only_call_has_no_image(self): + client = _client_returning({"verdict": "same", "confidence": 0.9}) + checker = vision_extract.make_same_meet_checker(client, "m") + checker(self._meet("X"), self._meet("X")) + # The same-meet check is text-only — no image attached. + assert "images" not in client.generate.call_args.kwargs + + def test_garbage_response_is_unknown(self): + client = _client_returning("not json") + checker = vision_extract.make_same_meet_checker(client, "m") + verdict = checker(self._meet("X"), self._meet("Y")) + assert verdict.verdict == "unknown" + assert verdict.confidence == 0.0 + + def test_bad_verdict_string_is_unknown(self): + client = _client_returning({"verdict": "maybe", "confidence": 0.5}) + checker = vision_extract.make_same_meet_checker(client, "m") + verdict = checker(self._meet("X"), self._meet("Y")) + assert verdict.verdict == "unknown" + + def test_response_error_degrades_to_unknown(self): + # A model error in the same-meet check must NOT abort the parse — + # degrade to "unknown" so the page carries forward for review. + import ollama + client = MagicMock() + client.generate.side_effect = ollama.ResponseError("boom", 500) + checker = vision_extract.make_same_meet_checker(client, "m") + verdict = checker(self._meet("X"), self._meet("Y")) + assert verdict.verdict == "unknown" + assert verdict.confidence == 0.0 + + +class TestCacheMore: def test_cache_ignored_when_model_differs(self, tmp_path): cache = tmp_path / "scan.raw.json" cache.write_text(json.dumps({