Skip to content

Retire the legacy prose-JSON repair stack - #43

Open
Ndevu12 wants to merge 1 commit into
mainfrom
refactor/retire-json-repair-stack
Open

Retire the legacy prose-JSON repair stack#43
Ndevu12 wants to merge 1 commit into
mainfrom
refactor/retire-json-repair-stack

Conversation

@Ndevu12

@Ndevu12 Ndevu12 commented Sep 3, 2026

Copy link
Copy Markdown
Owner

Summary

Native structured outputs (#38) made the prose-JSON repair machinery redundant: schemas are attached to agents via pydantic-ai output_type, validated natively, and retried with validation feedback by the framework itself. This change completes that migration by removing the parallel legacy path entirely — −5,307 / +150 lines — leaving one LLM call path (run_structured / try_run_structured) with the existing heuristic fallbacks as the safety net.

What was removed

  • src/utils/ repair stack (8 modules, ~4,400 lines): enhanced_response_handler, fallback_processing, content_quality, model_adaptation, enhanced_validation, quality_monitor, retry_manager, response_models, plus the repair-narrative message formatters that only they used.
  • src/retrieval/helpers_modules/ (~700 lines): JSON extraction/validation/recovery helpers whose only remaining consumers were the stack itself and query expansion's legacy branch. (src/retrieval/helpers.py is unrelated and stays.)
  • Legacy branches in synthesis (per-paper extraction and collective pass), gap analysis, and query expansion, along with the handler/session_id plumbing threaded through extract_papers, synthesize_collective, run_synthesis, analyze_gaps, SynthesisStage, and GapAnalysisStage.
  • Configuration: the llm.structured_outputs flag (settings, config/default.yaml, .env.example, docs) and the now-unused extraction_max_retries / collective_max_retries synthesis knobs. Research loop and verification now gate on synthesis.llm_enabled alone.
  • Tests: the deleted stack's suite (test_graceful_response_handling.py); synthesis and structured-output tests now mock src.models.structured.try_run_structured directly, preserving the original behavioral assertions (identity pinning, circuit breaker, max_llm_papers cap, heuristic fallbacks, call-count checks).

What did not change

  • Heuristic fallbacks at every LLM call site — a failed structured call still degrades gracefully, per paper and per stage.
  • The circuit breaker, max_llm_papers cap, concurrency control, and identity pinning (paper ID/title always come from retrieval metadata).
  • Ranking, retrieval, and evaluation behavior: golden-set metrics are byte-identical to main (verified by running the harness on both).

Verification

  • pytest: 411 passed, 1 skipped (count drops from 442 because the deleted stack's 31-test suite is gone)
  • ruff check src tests setups: clean
  • scripts/check_docs_policy.py: OK (63 nav pages, 6 hubs)
  • Golden-set harness: R@5 0.93, R@10 1.00, nDCG@10 0.97, MRR 1.00, citation validity 92% — identical on main and this branch

Native structured outputs (schema attached via pydantic-ai output_type)
are now the only LLM call path, so the parallel machinery that repaired
free-form JSON responses is removed end to end:

- Delete src/utils/{enhanced_response_handler, fallback_processing,
  content_quality, model_adaptation, enhanced_validation, quality_monitor,
  retry_manager, response_models} and src/retrieval/helpers_modules/
  (~5,100 lines), plus their repair-narrative message formatters.
- Strip the legacy branches and handler/session plumbing from synthesis,
  gap analysis, and query expansion; stages now call try_run_structured
  directly with heuristic fallbacks unchanged.
- Remove the llm.structured_outputs flag (settings, default.yaml,
  .env.example, docs) and the now-unused synthesis retry knobs; research
  loop and verification gate on synthesis.llm_enabled alone.
- Rework synthesis/structured-output tests to mock try_run_structured;
  drop the deleted stack's test suite; refresh affected docs.

Verified: 411 tests pass, ruff clean, docs policy OK, golden-set metrics
unchanged against main (R@5 0.93, nDCG@10 0.97, MRR 1.00, validity 92%).
@Ndevu12 Ndevu12 self-assigned this Sep 4, 2026
@Ndevu12 Ndevu12 added the enhancement New feature or request label Sep 4, 2026
@Ndevu12
Ndevu12 force-pushed the refactor/retire-json-repair-stack branch from 34e8228 to c47b842 Compare September 4, 2026 18:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant