Skip to content

Enable strict tool use on the Anthropic structured-output call - #99

Merged
exactml merged 2 commits into
masterfrom
fix/anthropic-strict-tool-use
Sep 20, 2026
Merged

exactml merged 2 commits into
masterfrom
fix/anthropic-strict-tool-use

Conversation

@exactml

@exactml exactml commented Sep 20, 2026

Copy link
Copy Markdown
Owner

Summary

  • AnthropicProvider.generate_structured now sets "strict": true on its tool definition
  • Anthropic's strict tool use compiles input_schema into a grammar and constrains token sampling at generation time, so the response is guaranteed schema-conformant -- not just requested to be. This is the server-side fix for the failure class ISSUE-96's retry was mitigating client-side: an extra wrapper key, a double-encoded string, a stringified array -- all observed live on _FindingsResponse, all a symptom of the model's free-form generation drifting from the schema
  • Verified both _FindingsResponse and its nested Finding already emit "additionalProperties": false at every level (a strict mode expectation, per Anthropic's docs examples) via pydantic's existing ConfigDict(extra="forbid") -- no schema changes needed, only the flag
  • The two-attempt retry from Retry once on a structured-output schema mismatch instead of failing outright #96/Retry once on a structured-output schema mismatch #97 stays in place, unchanged, as a dormant safety net -- kept per instruction, not removed. If _FindingsResponse ever grows complex enough to hit strict mode's documented compilation-time limits, that fails at the request level (anthropic.APIError, already handled, not retried) rather than reintroducing a validation mismatch, so the retry branch should simply go unused rather than get in the way

Relates to #96 -- not a new issue, this is the follow-up fix for the same root problem.

Test plan

  • make lint -- clean
  • make test -- 160 passed (1 new), asserting strict: True is actually present in the request sent to messages.create
  • Real-world verification: once released, re-run the review check on Resolve a diff's changed functions to their out-of-diff callers #94 (which has repeatedly hit this exact failure) to confirm it now comes back clean

exactml and others added 2 commits September 20, 2026 14:45
Relates to ISSUE-96 -- the retry added there was a client-side
mitigation for a server-side gap. strict: true closes that gap at
the source via grammar-constrained decoding, so the schema-mismatch
failures the retry exists to catch should no longer occur. The
retry stays in place as a dormant safety net.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 marginal review

PR #99: Enable strict tool use on the Anthropic structured-output call · open · 5144f395166e65483d632dc93bfe8fed207eb597 → 341e1765212417684125dd2f0298255680e2a54b · 3 files changed

Changed files (3)
  • CHANGELOG.md
  • marginal/providers/anthropic.py
  • tests/test_providers.py

@exactml
exactml merged commit 11a4494 into master Sep 20, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant