Repository navigation
Conversation
parse_llm_json could return None - or the wrong value - for answers that were perfectly well formed. Observed while running Claude CLI and Codex CLI as 'provider: cli' models against the codereview tool: the model produced a full review, the parser threw it away, and the model was reported as 'Failed to parse LLM response as JSON' and contributed nothing. Three shapes, three fixes, all additive: 1. Valid JSON was still sent through the repair heuristics. The Claude CLI --output-format json envelope is valid JSON whose 'result' field contains an escaped ```json block; the greedy fence stripper matched into that escaped block and shredded the envelope. Now a plain json.loads() runs first. 2. _strip_code_fences is greedy (first ``` to last ```) and does not match at all when prose follows the closing fence. _try_fenced_blocks() now tries each fenced block on its own, newest first, as a last resort at both exits. 3. When there is no clean JSON candidate, complete fenced blocks are preferred over scavenging with _extract_first_json_block. Previously a ```python block could yield a JSON-looking fragment, so the function 'succeeded' with a list and the real ```json block was never read. Also accept schema aliases in codereview: models diverge on the issue-list key. On one review task Claude CLI returned 'issues_found' on one run and 'verified_findings' on another, while Codex CLI returned 'findings' and no 'status' field. All carry the same id/severity/location dicts, so the aliases are read only after the original conditions fail - otherwise those models silently report 0 issues. Tests: 6 new cases in tests/unit/test_json_parser_llm_shapes.py. Three fail on main (the envelope returns None; the code-block shape returns a list instead of a dict; the alias helper does not exist) and the rest guard shapes that already worked. Full suite: 576 passed, ruff clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
parse_llm_jsoncan returnNone— or the wrong value — for answers that areperfectly well formed. I hit this while running Claude CLI and Codex CLI as
provider: climodels against thecodereviewtool: the model produced acomplete review, the parser discarded it, and the model was reported as
Failed to parse LLM response as JSONand contributed nothing to the review.Three distinct output shapes trigger it. All three are additive fixes.
1. Valid JSON was still sent through the repair heuristics
The Claude CLI
--output-format jsonenvelope is valid JSON whoseresultfield contains an escaped
```jsonblock._strip_code_fencesis greedy, soit matched into that escaped block and shredded the envelope — a plain
json.loads()on the same input succeeds.Fix: try
json.loads(text)first. If the input already parses, nothing needsrepairing.
2. Fenced blocks were never tried individually
_strip_code_fencesmatches from the first```to the last one, and doesnot match at all when prose follows the closing fence. A model that writes an
analysis containing a code block and then emits its JSON at the end defeats
both paths.
Fix:
_try_fenced_blocks()tries each fenced block on its own, newest first,as a last resort at both exit points.
3. A fragment could be scavenged out of a non-JSON code block
When there was no clean JSON candidate,
_extract_first_json_blockcould pull aJSON-looking fragment out of a
```pythonblock and "succeed" with it —usually returning a
list— so the real```jsonblock was never read. Thisone is the nastiest, because it fails silently with a plausible value rather
than returning
None.Fix: prefer complete fenced blocks over scavenging.
Also: schema aliases in
codereviewModels do not reliably honour the requested key. On a single review task I
measured Claude CLI returning
issues_foundon one run andverified_findingson another, and Codex CLI returning
findingswith nostatusfield at all.All three carry the same shape — a list of dicts with
id/severity/location— so
_extract_issue_list()accepts the aliases, but only after the originalconditions have been tried. Without this, those models report
0 issueswhiletheir text contains every finding.
Tests
tests/unit/test_json_parser_llm_shapes.py, 6 cases.Three fail on
main:test_valid_json_envelope_containing_a_fenced_block_is_returned_untouched— returnsNonetest_code_block_before_the_json_block—AssertionError: expected dict, got listtest_extract_issue_list_accepts_schema_aliases— helper does not existThe other three guard shapes that already work and must keep working.
Full suite: 576 passed, 1 skipped.
ruff checkclean on all touched files.Notes