Roadmap Phase 4: iterative research loop and claim verification

.env.example

@@ -113,5 +113,12 @@ RA_PIPELINE__STREAM_PROGRESS=true
 # RA_FULLTEXT__ENABLED=false
 # RA_FULLTEXT__MAX_PAPERS=5
 
+# Coverage-driven retrieval refinement (follow-up queries when coverage is thin)
+# RA_RESEARCH_LOOP__ENABLED=false
+# RA_RESEARCH_LOOP__MAX_ITERATIONS=1
+
+# Claim verification before report generation
+# RA_VERIFICATION__ENABLED=false
+
 # Optional: override config directory
 # RA_CONFIG_DIR=/path/to/config

config/default.yaml

@@ -95,13 +95,25 @@ pipeline:
     snowball: true
     rerank: true
     relevance_scoring: true
+    research_loop: true
     fulltext: true
     clustering: true
     synthesis: true
     gap_analysis: true
+    verification: true
     citation_export: true
     report_generation: true
 
+research_loop:
+  enabled: true
+  max_iterations: 1
+  min_sufficient_papers: 6
+  max_follow_up_queries: 3
+
+verification:
+  enabled: true
+  min_term_coverage: 0.5
+
 fulltext:
   enabled: true
   max_papers: 5

docs/architecture/stages/research-loop.md

@@ -0,0 +1,48 @@
+# Research Loop (Coverage-Driven Refinement)
+
+The research loop runs between [relevance scoring](relevance-scoring.md)
+and [fulltext](fulltext.md). It turns the one-shot pipeline into the
+assess–refine–stop cycle deep-research agents use: after filtering, the
+stage judges whether the kept papers actually answer the query; when
+coverage is thin it generates targeted follow-up queries, retrieves for
+them, and merges the new candidates through the same deduplication,
+ranking, and relevance machinery — repeating up to a configured budget.
+
+## Coverage assessment
+
+Two assessors, chosen by the LLM feature resolution:
+
+- **LLM** (when synthesis LLM mode is on and structured outputs are
+  enabled) — a structured verdict: `sufficient`, `missing_aspects`, and
+  `follow_up_queries` proposed as concise academic search phrases.
+- **Heuristic** (always available) — coverage requires at least
+  `min_sufficient_papers` kept papers *and* every core query concept
+  matched by at least one paper; follow-up queries combine each missing
+  concept with the covered ones.
+
+The final assessment is stored in the `coverage_assessment` artifact, so
+debug dumps show why the loop did or did not fire.
+
+## Refinement round
+
+Follow-up queries go through the standard retrieval fan-out, and the new
+candidates merge with the current set through deduplication (with
+cross-provider reconciliation), ranking, and the relevance filter — no
+special-case scoring path. Embedding-backend failures fall back to
+keyword ranking exactly as the ranking stage does.
+
+## Configuration
+
+| Key | Default | Meaning |
+|-----|---------|---------|
+| `research_loop.enabled` | `true` | Toggle the stage |
+| `research_loop.max_iterations` | `1` | Refinement rounds per run |
+| `research_loop.min_sufficient_papers` | `6` | Heuristic floor for sufficiency |
+| `research_loop.max_follow_up_queries` | `3` | Queries retrieved per round |
+
+## Failure behavior
+
+Retrieval failures during a refinement round pass the current ranking
+through with a warning; an assessment that never reaches sufficiency
+stops at the iteration budget. Stage metrics record iterations run, the
+follow-up queries used, and whether coverage ended sufficient.

docs/architecture/stages/verification.md

@@ -0,0 +1,45 @@
+# Verification (Claim Checking)
+
+The verification stage runs between [gap analysis](gap-analysis.md) and
+[citation export](citation-export.md). Before a report is generated,
+each paper's key points are checked against that paper's own sources —
+its abstract plus any grounded full-text passages from the
+[fulltext stage](fulltext.md). Unsupported claims are flagged rather
+than presented as fact.
+
+## What readers see
+
+- Flagged key points render with an _(unverified)_ marker in the
+  thematic findings.
+- The report's summary area carries an aggregate line, e.g.
+  *Claim verification (heuristic): 11/12 key points supported by paper
+  sources* — the `verification` field of the JSON report holds the same
+  numbers (`claims_checked`, `claims_unverified`, `method`).
+
+## How claims are checked
+
+- **LLM** (when synthesis LLM mode is on and structured outputs are
+  enabled) — a structured verdict per paper naming the claim numbers the
+  sources do not support, judged strictly from the provided material.
+- **Heuristic** (always available) — a claim is flagged when fewer than
+  `min_term_coverage` of its content terms appear in the paper's
+  sources. Flagging is deliberately conservative: with no sources at
+  all, nothing is flagged, and abstract-derived heuristic findings pass
+  by construction.
+
+Identity note: verification never rewrites claims — it only annotates
+`unverified_points` on the paper analyses and records the
+`verification_summary` artifact consumed by report generation.
+
+## Configuration
+
+| Key | Default | Meaning |
+|-----|---------|---------|
+| `verification.enabled` | `true` | Toggle the stage |
+| `verification.min_term_coverage` | `0.5` | Heuristic support threshold |
+
+## Failure behavior
+
+A failed LLM verdict falls back to the heuristic for that paper. The
+stage passes gap-analysis output through unchanged, so downstream
+stages are unaffected by verification results.

docs/configuration/environment-variables.md

@@ -122,7 +122,7 @@ Full weight list: [YAML reference](yaml-reference.md#ranking).
 | `RA_PIPELINE__SYNTHESIS_TIMEOUT_SECONDS` | `600` | Synthesis stage timeout |
 | `RA_PIPELINE__ENABLED_STAGES__<STAGE>` | `true` | Disable individual pipeline stages |
 
-Stage names: `query_understanding`, `query_expansion`, `retrieval`, `deduplication`, `ranking`, `snowball`, `rerank`, `relevance_scoring`, `fulltext`, `clustering`, `synthesis`, `gap_analysis`, `citation_export`, `report_generation`. See [Stage toggles](stage-toggles.md).
+Stage names: `query_understanding`, `query_expansion`, `retrieval`, `deduplication`, `ranking`, `snowball`, `rerank`, `relevance_scoring`, `research_loop`, `fulltext`, `clustering`, `synthesis`, `gap_analysis`, `verification`, `citation_export`, `report_generation`. See [Stage toggles](stage-toggles.md).
 
 ---
 
@@ -141,8 +141,12 @@ Stage names: `query_understanding`, `query_expansion`, `retrieval`, `deduplicati
 | `RA_FULLTEXT__MAX_PAPERS` | `5` | Top papers attempted per run |
 | `RA_FULLTEXT__MAX_PDF_MB` | `15` | Per-PDF download size cap |
 | `RA_FULLTEXT__TOP_CHUNKS_PER_PAPER` | `3` | Passages retrieved per paper |
+| `RA_RESEARCH_LOOP__ENABLED` | `true` | Coverage-driven retrieval refinement |
+| `RA_RESEARCH_LOOP__MAX_ITERATIONS` | `1` | Refinement rounds per run |
+| `RA_VERIFICATION__ENABLED` | `true` | Claim verification before reporting |
+| `RA_VERIFICATION__MIN_TERM_COVERAGE` | `0.5` | Heuristic support threshold |
 
-Full key lists live on the stage pages: [Snowball](../architecture/stages/snowball.md), [Rerank](../architecture/stages/rerank.md), [Fulltext](../architecture/stages/fulltext.md).
+Full key lists live on the stage pages: [Snowball](../architecture/stages/snowball.md), [Rerank](../architecture/stages/rerank.md), [Fulltext](../architecture/stages/fulltext.md), [Research loop](../architecture/stages/research-loop.md), [Verification](../architecture/stages/verification.md).
 
 ---
 

mkdocs.yml

@@ -79,10 +79,12 @@ nav:
           - architecture/stages/snowball.md
           - architecture/stages/rerank.md
           - architecture/stages/relevance-scoring.md
+          - architecture/stages/research-loop.md
           - architecture/stages/fulltext.md
           - architecture/stages/clustering.md
           - architecture/stages/synthesis.md
           - architecture/stages/gap-analysis.md
+          - architecture/stages/verification.md
           - architecture/stages/citation-export.md
           - architecture/stages/report-generation.md
   - Configuration:

src/analysis/verification.py

@@ -0,0 +1,182 @@
+# -*- coding: utf-8 -*-
+"""Claim verification against paper sources before report generation.
+
+Each paper's key points are checked against that paper's own sources —
+abstract plus any grounded full-text passages. Unsupported claims are
+flagged in the report rather than presented as fact, and the report
+carries an aggregate verification summary.
+
+Verification is LLM-backed (structured output) when synthesis LLM mode is
+on, with a term-coverage heuristic otherwise. Flagging is deliberately
+conservative: a claim is marked unverified only when its content terms
+are largely absent from the paper's sources.
+"""
+
+from __future__ import annotations
+
+import time
+
+from pydantic import BaseModel, Field
+
+from ..core.context import PipelineContext, StageResult
+from ..models.base import AgentRole
+from ..research.text_utils import extract_query_terms, term_matches_text
+from ..retrieval.models import (
+    GapAnalysisResult,
+    PaperAnalysis,
+    RankedPaper,
+    VerificationSummary,
+)
+
+VERIFICATION_ARTIFACT = "verification_summary"
+
+
+class _ClaimVerdicts(BaseModel):
+    """Structured output: which numbered claims lack support in the sources."""
+
+    unsupported_claim_numbers: list[int] = Field(default_factory=list)
+
+
+def _paper_sources(
+    analysis: PaperAnalysis,
+    abstracts: dict[str, str],
+    passages: dict[str, list[str]],
+) -> str:
+    parts: list[str] = []
+    if analysis.paper_id:
+        if analysis.paper_id in abstracts:
+            parts.append(abstracts[analysis.paper_id])
+        parts.extend(passages.get(analysis.paper_id, []))
+    return "\n".join(parts).lower()
+
+
+def heuristic_unsupported(
+    claims: list[str],
+    sources: str,
+    *,
+    min_term_coverage: float,
+) -> list[str]:
+    """Claims whose content terms are largely absent from the sources."""
+    if not sources:
+        return []
+
+    unsupported: list[str] = []
+    for claim in claims:
+        terms = extract_query_terms(claim)
+        if not terms:
+            continue
+        matched = sum(1 for term in terms if term_matches_text(term, sources))
+        if matched / len(terms) < min_term_coverage:
+            unsupported.append(claim)
+    return unsupported
+
+
+def _verification_prompt(analysis: PaperAnalysis, sources: str) -> str:
+    numbered = "\n".join(
+        f"{index}. {claim}" for index, claim in enumerate(analysis.key_points, start=1)
+    )
+    return (
+        f"Paper: {analysis.title}\n\n"
+        f"Source material (abstract and full-text passages):\n{sources[:4000]}\n\n"
+        f"Claims made about this paper:\n{numbered}\n\n"
+        "Which claim numbers are NOT supported by the source material?"
+    )
+
+
+async def _llm_unsupported(
+    analysis: PaperAnalysis,
+    sources: str,
+    ctx: PipelineContext,
+) -> list[str] | None:
+    from ..models.structured import try_run_structured
+
+    verdicts = await try_run_structured(
+        AgentRole.VERIFICATION,
+        _verification_prompt(analysis, sources),
+        _ClaimVerdicts,
+        ctx.config.llm,
+    )
+    if verdicts is None:
+        return None
+    return [
+        analysis.key_points[number - 1]
+        for number in verdicts.unsupported_claim_numbers
+        if 1 <= number <= len(analysis.key_points)
+    ]
+
+
+class VerificationStage:
+    """Pipeline stage that flags unsupported claims before reporting."""
+
+    name = "verification"
+
+    async def run(
+        self,
+        ctx: PipelineContext,
+        data: GapAnalysisResult,
+    ) -> StageResult[GapAnalysisResult]:
+        started = time.perf_counter()
+        config = ctx.config.verification
+
+        def _result(summary: VerificationSummary | None) -> StageResult[GapAnalysisResult]:
+            metrics: dict[str, object] = {}
+            if summary is not None:
+                ctx.set_artifact(VERIFICATION_ARTIFACT, summary.model_dump())
+                metrics = {
+                    "claims_checked": summary.claims_checked,
+                    "claims_unverified": summary.claims_unverified,
+                    "method": summary.method,
+                }
+            return StageResult(
+                output=data,
+                duration_ms=(time.perf_counter() - started) * 1000,
+                metrics=metrics,
+            )
+
+        analyses: list[PaperAnalysis] = ctx.get_artifact("paper_analyses") or []
+        if not config.enabled or not analyses:
+            return _result(None)
+
+        ranked: list[RankedPaper] = ctx.get_artifact("ranked_papers") or []
+        abstracts = {
+            item.paper.paper_id: item.paper.abstract or "" for item in ranked
+        }
+        passages: dict[str, list[str]] = ctx.get_artifact("fulltext_passages") or {}
+
+        use_llm = ctx.config.synthesis.llm_enabled and ctx.config.llm.structured_outputs
+        method = "llm" if use_llm else "heuristic"
+
+        checked = 0
+        unverified_total = 0
+        verified_analyses: list[PaperAnalysis] = []
+
+        for analysis in analyses:
+            sources = _paper_sources(analysis, abstracts, passages)
+            claims = analysis.key_points
+            checked += len(claims)
+
+            unsupported: list[str] | None = None
+            if use_llm and claims and sources:
+                unsupported = await _llm_unsupported(analysis, sources, ctx)
+            if unsupported is None:
+                method = "heuristic" if not use_llm else method
+                unsupported = heuristic_unsupported(
+                    claims,
+                    sources,
+                    min_term_coverage=config.min_term_coverage,
+                )
+
+            unverified_total += len(unsupported)
+            verified_analyses.append(
+                analysis.model_copy(update={"unverified_points": unsupported})
+                if unsupported
+                else analysis
+            )
+
+        ctx.set_artifact("paper_analyses", verified_analyses)
+        summary = VerificationSummary(
+            claims_checked=checked,
+            claims_unverified=unverified_total,
+            method=method,
+        )
+        return _result(summary)

src/config/settings.py

@@ -151,6 +151,22 @@ class RetrievalConfig(BaseModel):
     )
 
 
+class ResearchLoopConfig(BaseModel):
+    """Iterative coverage-driven retrieval refinement."""
+
+    enabled: bool = True
+    max_iterations: int = 1
+    min_sufficient_papers: int = 6
+    max_follow_up_queries: int = 3
+
+
+class VerificationConfig(BaseModel):
+    """Claim verification against paper sources before report generation."""
+
+    enabled: bool = True
+    min_term_coverage: float = 0.5
+
+
 class FulltextConfig(BaseModel):
     """Open-access full-text grounding for top-ranked papers."""
 
@@ -200,10 +216,12 @@ class PipelineConfig(BaseModel):
             "snowball": True,
             "rerank": True,
             "relevance_scoring": True,
+            "research_loop": True,
             "fulltext": True,
             "clustering": True,
             "synthesis": True,
             "gap_analysis": True,
+            "verification": True,
             "citation_export": True,
             "report_generation": True,
         }
@@ -280,6 +298,8 @@ class AppSettings(BaseSettings):
     snowball: SnowballConfig = Field(default_factory=SnowballConfig)
     rerank: RerankConfig = Field(default_factory=RerankConfig)
     fulltext: FulltextConfig = Field(default_factory=FulltextConfig)
+    research_loop: ResearchLoopConfig = Field(default_factory=ResearchLoopConfig)
+    verification: VerificationConfig = Field(default_factory=VerificationConfig)
     pipeline: PipelineConfig = Field(default_factory=PipelineConfig)
     memory: MemoryConfig = Field(default_factory=MemoryConfig)
     synthesis: SynthesisConfig = Field(default_factory=SynthesisConfig)