Skip to content

Constrain vision output with a JSON Schema so per-field confidences survive - #47

Merged
gavinbee merged 1 commit into
mainfrom
45-vision-confidence-structured-outputs
May 30, 2026
Merged

Constrain vision output with a JSON Schema so per-field confidences survive#47
gavinbee merged 1 commit into
mainfrom
45-vision-confidence-structured-outputs

Conversation

@gavinbee

Copy link
Copy Markdown
Collaborator

Summary

Fixes the all-0.5 confidences in vision-path output (#45).

The vision call used format="json", which only guarantees valid JSON — not our nested {value, confidence} shape. Smaller models (qwen2.5vl:3b) collapse each field to a bare scalar, so _parse_field defaulted every confidence to _DEFAULT_CONFIDENCE (0.5), silently destroying the signal that drives low-confidence review.

  • Build a JSON Schema from the canonical src/schema constants (_build_json_schema) and pass it to Ollama's format= so decoding is constrained to the object shape. Schema is generated from the same field lists as the prompt skeleton, so it can't drift.
  • Detect an all-flat-scalar page (_response_is_flat) and log a warning instead of masking it — so this failure mode is visible if a model still ignores the schema.
  • Document the model floor in docs/models.md: confidence quality is model-dependent — 3B is unreliable (constant value), 7B is the practical floor for confidence-driven features.

Investigation notes

  • 7B already respected the object shape under plain format="json"; with the schema its output is unchanged ({1.0, 0.9, 0.6} spread, 0.6 landing on ambiguous successful cells) — no regression.
  • 3B under plain format="json" returned flat scalars → all 0.5. With the schema it emits the object shape (though its confidences flatline — a model limitation the schema can't fix, hence the doc note).

Test plan

  • pytest -q — 286 passed
  • 7B end-to-end (--no-cache) — no content/confidence regression
  • New tests: TestJsonSchema (schema built from canonical fields), TestFlatResponseWarning (flat detection + warning), updated format assertion

Closes #45

🤖 Generated with Claude Code

…urvive

The vision call used format="json", which only guarantees valid JSON, not
our nested {value, confidence} shape. Smaller models (qwen2.5vl:3b) collapse
each field to a bare scalar, so _parse_field defaulted every confidence to
0.5 — silently destroying the signal that drives low-confidence review.

Build a JSON Schema from the canonical src/schema constants and pass it to
Ollama's format= so decoding is constrained to the object shape. Also detect
an all-flat-scalar page and warn instead of masking it. Document that
confidence quality is model-dependent (3b unreliable, 7b is the floor).

Verified: 7b output unchanged (no regression); 3b now emits the object shape.

Closes #45

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

@gavinbee gavinbee left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gavinbee
gavinbee merged commit af3ffb9 into main May 30, 2026
1 check passed
@gavinbee
gavinbee deleted the 45-vision-confidence-structured-outputs branch May 30, 2026 04:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Vision extraction confidences are always 0.5 (model returns flat scalars, not {value, confidence} objects)

1 participant