Skip to content

fix(json_parser): keep valid JSON intact and read fenced blocks reliably - #11

Open
fhrddn wants to merge 1 commit into
religa:mainfrom
fhrddn:fix/robust-json-parsing-and-issue-keys
Open

fhrddn wants to merge 1 commit into
religa:mainfrom
fhrddn:fix/robust-json-parsing-and-issue-keys

Conversation

@fhrddn

@fhrddn fhrddn commented Sep 8, 2026

Copy link
Copy Markdown

Problem

parse_llm_json can return None — or the wrong value — for answers that are
perfectly well formed. I hit this while running Claude CLI and Codex CLI as
provider: cli models against the codereview tool: the model produced a
complete review, the parser discarded it, and the model was reported as
Failed to parse LLM response as JSON and contributed nothing to the review.

Three distinct output shapes trigger it. All three are additive fixes.

1. Valid JSON was still sent through the repair heuristics

The Claude CLI --output-format json envelope is valid JSON whose result
field contains an escaped ```json block. _strip_code_fences is greedy, so
it matched into that escaped block and shredded the envelope — a plain
json.loads() on the same input succeeds.

Fix: try json.loads(text) first. If the input already parses, nothing needs
repairing.

2. Fenced blocks were never tried individually

_strip_code_fences matches from the first ``` to the last one, and does
not match at all when prose follows the closing fence. A model that writes an
analysis containing a code block and then emits its JSON at the end defeats
both paths.

Fix: _try_fenced_blocks() tries each fenced block on its own, newest first,
as a last resort at both exit points.

3. A fragment could be scavenged out of a non-JSON code block

When there was no clean JSON candidate, _extract_first_json_block could pull a
JSON-looking fragment out of a ```python block and "succeed" with it —
usually returning a list — so the real ```json block was never read. This
one is the nastiest, because it fails silently with a plausible value rather
than returning None.

Fix: prefer complete fenced blocks over scavenging.

Also: schema aliases in codereview

Models do not reliably honour the requested key. On a single review task I
measured Claude CLI returning issues_found on one run and verified_findings
on another, and Codex CLI returning findings with no status field at all.
All three carry the same shape — a list of dicts with id/severity/location
— so _extract_issue_list() accepts the aliases, but only after the original
conditions have been tried. Without this, those models report 0 issues while
their text contains every finding.

Tests

tests/unit/test_json_parser_llm_shapes.py, 6 cases.

Three fail on main:

  • test_valid_json_envelope_containing_a_fenced_block_is_returned_untouched — returns None
  • test_code_block_before_the_json_block — AssertionError: expected dict, got list
  • test_extract_issue_list_accepts_schema_aliases — helper does not exist

The other three guard shapes that already work and must keep working.

Full suite: 576 passed, 1 skipped. ruff check clean on all touched files.

Notes

  • No behaviour is removed: every new path runs only after the existing ones fail.
  • Happy to split this into two PRs (parser vs. codereview) if you prefer.

parse_llm_json could return None - or the wrong value - for answers that were
perfectly well formed. Observed while running Claude CLI and Codex CLI as
'provider: cli' models against the codereview tool: the model produced a full
review, the parser threw it away, and the model was reported as
'Failed to parse LLM response as JSON' and contributed nothing.

Three shapes, three fixes, all additive:

1. Valid JSON was still sent through the repair heuristics. The Claude CLI
   --output-format json envelope is valid JSON whose 'result' field contains an
   escaped ```json block; the greedy fence stripper matched into that escaped
   block and shredded the envelope. Now a plain json.loads() runs first.

2. _strip_code_fences is greedy (first ``` to last ```) and does not match at
   all when prose follows the closing fence. _try_fenced_blocks() now tries each
   fenced block on its own, newest first, as a last resort at both exits.

3. When there is no clean JSON candidate, complete fenced blocks are preferred
   over scavenging with _extract_first_json_block. Previously a ```python block
   could yield a JSON-looking fragment, so the function 'succeeded' with a list
   and the real ```json block was never read.

Also accept schema aliases in codereview: models diverge on the issue-list key.
On one review task Claude CLI returned 'issues_found' on one run and
'verified_findings' on another, while Codex CLI returned 'findings' and no
'status' field. All carry the same id/severity/location dicts, so the aliases
are read only after the original conditions fail - otherwise those models
silently report 0 issues.

Tests: 6 new cases in tests/unit/test_json_parser_llm_shapes.py. Three fail on
main (the envelope returns None; the code-block shape returns a list instead of
a dict; the alias helper does not exist) and the rest guard shapes that already
worked. Full suite: 576 passed, ruff clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant