Skip to content

Wire the vision path into the CLI (end-to-end scanned-PDF parsing) - #44

Merged
gavinbee merged 6 commits into
mainfrom
10-end-to-end-vision-cli
May 30, 2026
Merged

Wire the vision path into the CLI (end-to-end scanned-PDF parsing)#44
gavinbee merged 6 commits into
mainfrom
10-end-to-end-vision-cli

Conversation

@gavinbee

@gavinbee gavinbee commented May 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

main.py now picks a path by whether the PDF has fillable widgets:

  • Form-field path (unchanged): fast, deterministic, no model. Template defaults to swim_ontario_v1 or --template; no detection.
  • Vision path (new): resolve a vision model (--vision-model, else gpu_detect tier auto-pick) → start Ollama via the OllamaDaemon context manager (auto-pull unless --no-auto-pull) → detect the template from page 1 (unless --template) → extract each page with vision_extract (cached to <stem>.raw.json unless --no-cache) → merge with a vision-model-backed same_meet_checker → write.

Flags promoted to functional: --vision-model, --no-cache, --no-auto-pull. --template default changed to auto-detect (vision) / swim_ontario_v1 (form-field). Interactive-review flags stay reserved (#12–#14).

Exit codes: Ollama binary/model missing or vision parse failure → 1; template undetectable or recognized-but-stubbed province → 2; MultiMeetError → 4.

  • src/vision_extract.py: make_same_meet_checker(client, model) — a merge.SameMeetChecker backed by the already-loaded vision model (text-only, no image) so multi-page scans with OCR-noisy headers get a real judgement instead of a spurious MultiMeetError.
  • Branding: CLI docstring → "SwimBlocks Deck Eval Parser"; argparse prog → deck-eval-parser.

Test plan

  • pytest tests/ -q — 264 passing (250 prior + 14 new). New: TestVisionPath (9) covering happy path with override and with detection, model auto-pick, Ollama-binary-missing, detection error, detected-stub (Quebec), vision-extraction error, no-rows, --no-cache threading; make_same_meet_checker (5). All mock the daemon/detection/extraction — no model runs in tests.
  • CI goes green.
  • Real-model smoke test on the RTX 3070 against an actual scanned PDF (needs ollama pull qwen2.5vl:7b).

Closes #10.

🤖 Generated with Claude Code

@gavinbee @claude
Wire the vision path into the CLI (end-to-end scanned-PDF parsing)
e7d4a19 
main.py now picks a path by whether the PDF has fillable widgets:

- Form-field path (unchanged behavior): fast, deterministic, no model.
  Template defaults to swim_ontario_v1 or --template; no detection.

- Vision path (new): resolves a vision model (--vision-model or
  gpu_detect tier auto-pick), starts Ollama via the OllamaDaemon
  context manager (auto-pull unless --no-auto-pull), detects the
  template from page 1 (unless --template), extracts each page with
  vision_extract (cached to <stem>.raw.json unless --no-cache), merges
  with a vision-model-backed same_meet_checker, and writes outputs.

Flags promoted from reserved to functional: --vision-model, --no-cache,
--no-auto-pull. --template default changed from swim_ontario_v1 to None
(auto-detect on the vision path, swim_ontario_v1 fallback on the
form-field path). The interactive-review flags remain reserved (#12-14).

Error handling maps to exit codes: Ollama binary/model missing or vision
parse failure -> 1; template undetectable or a recognized-but-stubbed
province (NotImplementedError) -> 2; MultiMeetError -> 4.

- src/vision_extract.py: added make_same_meet_checker(client, model),
  a merge.SameMeetChecker backed by the already-loaded vision model
  (text-only prompt, no image) so multi-page scans whose headers differ
  only by OCR noise get a real same-meet judgement instead of a
  spurious MultiMeetError. Imports SameMeetVerdict lazily to avoid a
  load-order coupling with merge.

- Branding: CLI docstring is now "SwimBlocks Deck Eval Parser"; argparse
  prog is "deck-eval-parser" (matches the repo name). (The leftover
  gavinbee URLs in ollama_runtime / template_detect are being updated
  separately.)

- tests/test_main.py: replaced the stale "vision not implemented" test
  with a TestVisionPath class (9 tests) and updated argparse default
  tests. All mock the daemon / detection / extraction so no model runs.

- tests/test_vision_extract.py: 5 tests for make_same_meet_checker.

- docs/architecture.md, docs/usage.md, README.md: updated to reflect
  that both paths now run end-to-end.

Closes #10.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@gavinbee
gavinbee force-pushed the 10-end-to-end-vision-cli branch from 67c035c to e7d4a19 Compare May 29, 2026 03:36
gavinbee and others added 5 commits May 28, 2026 23:55
@gavinbee @claude
Log per-page progress and timing during vision extraction
eb45faf 
A real run on the RTX 3070 surfaced the gap: between "model pulled" and
the final summary the CLI went completely silent for the model cold-load
plus per-page inference, which reads as a hang.

extract_pdf now logs at INFO, per page:
- "Page N/total: extracting with <model> …" before the call
- "Page N/total: done in X.Xs (R row(s))" after, or
- "Page N/total: using cached vision response" on a cache hit

The first page's timing naturally includes the model's cold load into
VRAM, so the user can see why page 1 is slow and subsequent pages are
faster. Visible with -v.

tests/test_vision_extract.py: 2 new tests asserting the per-page
extract/done lines (with row count) and the cache-hit line.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@gavinbee @claude
Handle Ollama model errors gracefully (no more raw tracebacks)
9a81c8e 
A real run hit an Ollama 500 (GGML_ASSERT in the qwen2.5vl vision
encoder — VRAM pressure / model-build incompatibility on an 8 GB card)
during page extraction. We weren't catching ollama.ResponseError, so it
surfaced as a stack trace instead of a clean message + exit code.

- vision_extract.describe_model_error(): shared, actionable message for
  an Ollama server error — names the model + status, and suggests the
  smaller model (--vision-model qwen2.5vl:3b), updating Ollama, and
  freeing VRAM.

- _generate() now catches ollama.ResponseError and re-raises it as
  VisionExtractionError (exit 1, no traceback). Deterministic 500s
  aren't retried.

- make_same_meet_checker's checker degrades a ResponseError to an
  "unknown" verdict (confidence 0.0) with a warning, so a model hiccup
  during reconciliation doesn't abort an otherwise-good multi-page
  parse — the page just surfaces for review.

- template_detect.detect_template catches ResponseError and raises a
  clean TemplateDetectionError with the same guidance.

- tests: ResponseError → VisionExtractionError (with 3b suggestion, no
  retry), checker → unknown, detect_template → TemplateDetectionError.

This is error-handling only; it does not fix the underlying GGML assert
(an Ollama/model-runtime issue). The fix makes the failure legible and
points at the qwen2.5vl:3b workaround.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@gavinbee @claude
Cap rasterized image long edge to keep vision encoding tractable
b5d2a0e 
Real-run finding on an RTX 3070: the vision-path slowness (and likely
the 7B GGML projector assert) traces to image size, not the model. A
letter-landscape page at 200 DPI is ~2200 px on the long edge; Ollama's
qwen2.5vl vision tower often runs on CPU, so encoding an image that
large is a multi-minute CPU grind with the GPU idle. The user observed
exactly this — GPU spike (model load) then sustained CPU during the
in-flight template-detection generate call (the POST only logs once it
returns, which is why it looked like a hang before the request).

rasterize_page now downscales (LANCZOS, aspect preserved) so the longest
edge is <= DEFAULT_MAX_EDGE_PX (1600) by default; pass max_edge_px=0 to
disable. Both the vision extractor and template detection go through
rasterize_page, so both benefit. Form text stays legible at 1600 px.

tests: long-edge capped by default, cap disabled with 0, aspect ratio
preserved.

This is a mitigation, not a full fix — getting the vision encoder onto
the GPU (newer Ollama) is the larger lever and is environment-side.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@gavinbee @claude
Make model-call progress legible; stop httpx's after-the-fact noise
074e902 
The confusing part of a real run was that the only marker around the
slow template-detection call was httpx's "HTTP Request: POST /api/generate"
line — which httpx logs AFTER the response returns, so it reads like
"starting" when it actually means "finished". The long CPU grind
appeared to happen before the log, when really the log WAS the end of
the call.

- template_detect now logs "Detecting template from page 1 of X with
  <model> …" BEFORE the call and "Detected … in N.Ns" AFTER, mirroring
  the per-page extraction logs. So every model call has an explicit
  before/after at INFO.

- main._configure_logging now silences httpx/httpcore to WARNING unless
  -vv (DEBUG). Our own before/after logs carry the INFO story; httpx's
  post-response lines only show when the user explicitly wants full
  debug output.

- tests: httpx quieted at default/-v, allowed at -vv.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@gavinbee @claude
Point model-error guidance at the real Ollama qwen2.5vl regression
6d1e81f 
Diagnosing a real run turned up that both failures are documented Ollama
regressions, not VRAM or PDF issues:
- GGML_ASSERT(a->ne[2]*4==b->ne[0]) — Qwen2.5-VL CUDA crash regressed in
  Ollama >= 0.13.x (works on 0.12.x). ollama#13630
- qwen2.5vl:3b won't GPU-offload on 8 GB cards since 0.13.4, falling back
  to 100% CPU. ollama#13687

My earlier error message said "update Ollama to the latest version",
which is exactly wrong — latest HAS the regression; the fix is to
downgrade to 0.12.x.

- describe_model_error() now names the known regression, says it works
  on 0.12.x, and points at docs/troubleshooting.md (keeping the VRAM /
  smaller-model hint as the secondary possibility).
- docs/troubleshooting.md: filled in with both regressions, the
  downgrade workaround, issue links, how to check the Ollama version,
  and a note on why the tier picker still assumes a working runtime.
- test updated to assert the new guidance (0.12.x + troubleshooting
  pointer) instead of the old "update Ollama" wording.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@gavinbee

gavinbee commented May 30, 2026

Copy link
Copy Markdown
Collaborator Author

LGTM

@gavinbee

gavinbee commented May 30, 2026

Copy link
Copy Markdown
Collaborator Author

There is likely a bug with extraction confidence scores, since they are all 0.5 or 1.0. Merging since this does work end to end and extract something. Something >>>>> Nothing :)

@gavinbee
gavinbee merged commit 983a76c into main May 30, 2026
1 check passed
@gavinbee
gavinbee deleted the 10-end-to-end-vision-cli branch May 30, 2026 03:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

End-to-end vision CLI

1 participant

@gavinbee