Roadmap Phase 4: iterative research loop and claim verification

[Ndevu12]

Summary

Implements Phase 4 of the improvement roadmap — the final phase. The pipeline now behaves like a deep-research agent: it assesses whether retrieval actually answered the query and refines with follow-up searches when it did not, and it verifies report claims against their sources before publication, flagging what it cannot support instead of presenting it as fact.

Research loop (new stage: relevance scoring → research_loop → fulltext)

• Coverage assessment — a structured LLM verdict (sufficient / missing_aspects / follow_up_queries) when synthesis LLM mode is on; otherwise a concept-coverage heuristic: at least min_sufficient_papers kept papers and every core query concept matched by at least one paper. The assessment is stored in the coverage_assessment artifact for debug dumps.
• Refinement round — follow-up queries go through the standard retrieval fan-out, and new candidates merge via the existing deduplication (with cross-provider reconciliation), ranking, and relevance filter — no special-case scoring path — repeating up to research_loop.max_iterations (default 1).
• Multi-concept queries whose aspects went unmatched now trigger at least one targeted refinement round, per the roadmap's Phase 4 exit criteria.

Verification (new stage: gap analysis → verification → citation export)

• Each paper's key points are checked against that paper's own sources — abstract plus the grounded full-text passages from Phase 2. LLM mode issues a structured per-paper verdict naming unsupported claim numbers; the heuristic flags claims whose content terms are largely absent from the sources (deliberately conservative: no sources ⇒ no flags).
• Unsupported claims are flagged, never rewritten: paper analyses gain unverified_points, reports mark them (unverified) in the thematic findings, and both markdown and JSON reports carry an aggregate VerificationSummary (claims checked/unverified, method) — e.g. Claim verification (heuristic): 11/12 key points supported by paper sources.

Plumbing

Config sections research_loop and verification (pydantic + YAML in sync), stage toggles, progress labels, two new AgentRoles with both structured and legacy prompts, stage docs registered in the navigation, and env vars documented in .env.example plus the environment reference.

Verification

• pipenv run pytest tests/ — 441 passed (15 new tests: heuristic coverage sufficiency and follow-up generation, loop refinement/budget/failure paths, heuristic claim support and flagging, passage-extended sources, stage artifacts, and report rendering of unverified markers), 1 skipped
• ruff check src tests setups — clean; scripts/check_docs_policy.py — passes (63 nav pages)
• LLM-backed coverage and verification verdicts are exercised through the structured-outputs layer from Roadmap Phase 3: native structured outputs for all LLM call sites #38; real-model behavior needs the same local Ollama validation run noted there

🤖 Generated with Claude Code

https://claude.ai/code/session_01Jefz7XDSjxJ2KVYQgSSDpe

---

Generated by Claude Code