Roadmap P2: full-text grounding — open-access RAG with verbatim evidence

.env.example

@@ -16,6 +16,20 @@
 # RA_CROSSREF_MAILTO=you@example.com
 # CROSSREF_MAILTO=you@example.com
 
+# PubMed (NCBI E-utilities) — optional, raises rate limits
+# NCBI_API_KEY=your_ncbi_key_here
+
+# CORE — required when the core provider is enabled
+# (free keys: https://core.ac.uk/services/api)
+# CORE_API_KEY=your_core_key_here
+
+# Enable additional providers (openalex + semantic_scholar are on by default)
+# RA_RETRIEVAL__PROVIDERS__ARXIV__ENABLED=true
+# RA_RETRIEVAL__PROVIDERS__CROSSREF__ENABLED=true
+# RA_RETRIEVAL__PROVIDERS__PUBMED__ENABLED=true
+# RA_RETRIEVAL__PROVIDERS__DBLP__ENABLED=true
+# RA_RETRIEVAL__PROVIDERS__CORE__ENABLED=true
+
 # =============================================================================
 # LLM — Ollama (default, local-first)
 # =============================================================================
@@ -55,7 +69,7 @@ RA_LLM__API_KEY=ollama
 # =============================================================================
 
 # RA_LLM__PROVIDER=anthropic
-# RA_LLM__MODEL=claude-3-5-haiku-latest
+# RA_LLM__MODEL=claude-sonnet-5
 # ANTHROPIC_API_KEY=sk-ant-...
 # Or set the unified key:
 # RA_LLM__API_KEY=sk-ant-...
@@ -70,7 +84,29 @@ RA_RANKING__TOP_K=25
 RA_PIPELINE__DEBUG=false
 # Live stage + LLM progress on stderr (auto-disabled when stderr is not a TTY)
 RA_PIPELINE__STREAM_PROGRESS=true
-RA_DEBUG=1
+# Verbose debug mode (JSON debug dumps, DEBUG console logs)
+# RA_DEBUG=1
+
+# Console log level (files under logs/ always get full detail)
+# RA_CONSOLE_LOG_LEVEL=INFO
+
+# Skip the Ollama setup/health check on startup (CI, containers)
+# RA_SKIP_SETUP_CHECK=1
+
+# =============================================================================
+# Accuracy stages (snowball / rerank / fulltext) — on by default
+# =============================================================================
+
+# Citation-graph expansion of top-ranked papers
+# RA_SNOWBALL__ENABLED=false
+
+# Cross-encoder reranking of the top papers
+# RA_RERANK__ENABLED=false
+# RA_RERANK__MODEL=cross-encoder/ms-marco-MiniLM-L-6-v2
+
+# Open-access full-text grounding (PDF download, passage evidence)
+# RA_FULLTEXT__ENABLED=false
+# RA_FULLTEXT__MAX_PAPERS=5
 
 # Optional: override config directory
 # RA_CONFIG_DIR=/path/to/config

Pipfile

@@ -10,6 +10,7 @@ name = "pytorch-cpu"
 
 [packages]
 pydantic-ai = "*"
+pymupdf = {version = "*", index = "pypi"}
 aiohttp = "*"
 pydantic = "*"
 pytest = "*"

config/default.yaml

@@ -94,12 +94,23 @@ pipeline:
     snowball: true
     rerank: true
     relevance_scoring: true
+    fulltext: true
     clustering: true
     synthesis: true
     gap_analysis: true
     citation_export: true
     report_generation: true
 
+fulltext:
+  enabled: true
+  max_papers: 5
+  max_pdf_mb: 15
+  request_timeout_seconds: 30
+  cache_dir: data/fulltext
+  chunk_chars: 1400
+  chunk_overlap: 200
+  top_chunks_per_paper: 3
+
 rerank:
   enabled: true
   model: cross-encoder/ms-marco-MiniLM-L-6-v2

docs/architecture/stages/fulltext.md

@@ -0,0 +1,51 @@
+# Fulltext (Grounded Passages)
+
+The fulltext stage runs between [relevance scoring](relevance-scoring.md)
+and [clustering](clustering.md). For the top relevance-filtered papers it
+resolves open-access PDFs, downloads and parses them, chunks the text
+section-aware, and retrieves the passages most relevant to the query. The
+passages ground synthesis: extractions quote verbatim evidence, and
+reports show it under each paper as **Evidence (from full text)** — the
+difference between a summary generator and a research assistant whose
+claims can be checked.
+
+## Flow
+
+1. **Resolve** — metadata-first, no extra API calls: arXiv IDs map
+   directly to PDF URLs, OpenAlex work records already carry open-access
+   locations in raw metadata, and CORE results expose download URLs.
+   Closed-access papers simply resolve to nothing.
+2. **Download** — cached under `data/fulltext/` keyed by paper ID, with
+   size and content-type guards (`CachingPDFDownloader`).
+3. **Parse** — PyMuPDF text extraction with heuristic section detection;
+   the references section is truncated to keep bibliography noise out of
+   retrieval (`src/fulltext/parser.py`).
+4. **Chunk** — paragraphs packed into overlapping windows that carry
+   their section name (`SectionAwareChunker`).
+5. **Retrieve** — chunks are embedded through the existing embedding
+   provider and searched by cosine similarity, falling back to BM25 when
+   the embedding backend is unavailable (`InMemoryFulltextIndex`).
+
+The per-paper passages land in the `fulltext_passages` artifact.
+Synthesis includes them in extraction prompts (LLM mode asks for short
+verbatim quotes; heuristic mode attaches trimmed passages directly), and
+the markdown renderer prints the quotes under each paper.
+
+## Configuration
+
+| Key | Default | Meaning |
+|-----|---------|---------|
+| `fulltext.enabled` | `true` | Toggle the stage |
+| `fulltext.max_papers` | `5` | Top papers attempted per run |
+| `fulltext.max_pdf_mb` | `15` | Per-PDF download size cap |
+| `fulltext.request_timeout_seconds` | `30` | Per-download timeout |
+| `fulltext.cache_dir` | `data/fulltext` | PDF cache location |
+| `fulltext.chunk_chars` / `chunk_overlap` | `1400` / `200` | Chunk window sizing |
+| `fulltext.top_chunks_per_paper` | `3` | Passages retrieved per paper |
+
+## Failure behavior
+
+Grounding is best-effort by design: closed-access papers, failed
+downloads, malformed PDFs, or a missing `pymupdf` backend reduce coverage
+and never break the run. Stage metrics report PDFs resolved, downloaded,
+chunks indexed, and papers that ended up with passages.

docs/configuration/environment-variables.md

@@ -20,6 +20,8 @@ Boolean env values accept standard truthy strings (`true`, `1`, `yes`).
 |----------|---------|-------------|
 | `RA_CONFIG_DIR` | `config/` (project root) | Override directory for YAML files |
 | `RA_DEBUG` | unset | Alias for debug mode (`1`, `true`, `yes`) — OR-combined with `RA_PIPELINE__DEBUG` |
+| `RA_SKIP_SETUP_CHECK` | unset | Skip the Ollama setup/health check on startup (CI, containers) |
+| `RA_CONSOLE_LOG_LEVEL` | `WARNING` | Log level shown on the console; files always get full detail |
 
 ---
 
@@ -103,6 +105,8 @@ Full weight list: [YAML reference](yaml-reference.md#ranking).
 | `S2_API_KEY` | No | Semantic Scholar — higher rate limits when set |
 | `RA_CROSSREF_MAILTO` | Recommended | CrossRef polite pool (User-Agent mailto) |
 | `CROSSREF_MAILTO` | Recommended | Alias for CrossRef mailto |
+| `NCBI_API_KEY` | No | PubMed E-utilities — higher rate limits when set |
+| `CORE_API_KEY` | For `core` | Required when the CORE provider is enabled |
 
 ---
 
@@ -117,7 +121,27 @@ Full weight list: [YAML reference](yaml-reference.md#ranking).
 | `RA_PIPELINE__SYNTHESIS_TIMEOUT_SECONDS` | `600` | Synthesis stage timeout |
 | `RA_PIPELINE__ENABLED_STAGES__<STAGE>` | `true` | Disable individual pipeline stages |
 
-Stage names: `query_understanding`, `query_expansion`, `retrieval`, `deduplication`, `ranking`, `relevance_scoring`, `clustering`, `synthesis`, `gap_analysis`, `citation_export`, `report_generation`. See [Stage toggles](stage-toggles.md).
+Stage names: `query_understanding`, `query_expansion`, `retrieval`, `deduplication`, `ranking`, `snowball`, `rerank`, `relevance_scoring`, `fulltext`, `clustering`, `synthesis`, `gap_analysis`, `citation_export`, `report_generation`. See [Stage toggles](stage-toggles.md).
+
+---
+
+## Snowball, rerank, fulltext
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `RA_SNOWBALL__ENABLED` | `true` | Citation-graph expansion of top-ranked papers |
+| `RA_SNOWBALL__MAX_SEED_PAPERS` | `5` | Seeds for the one-hop expansion |
+| `RA_SNOWBALL__MAX_NEW_PAPERS` | `30` | Cap on candidates entering re-ranking |
+| `RA_RERANK__ENABLED` | `true` | Cross-encoder reranking of the top papers |
+| `RA_RERANK__MODEL` | `cross-encoder/ms-marco-MiniLM-L-6-v2` | Any sentence-transformers cross-encoder |
+| `RA_RERANK__TOP_N` | `25` | Papers scored by the cross-encoder |
+| `RA_RERANK__BLEND_WEIGHT` | `0.5` | Cross-encoder share of the final score |
+| `RA_FULLTEXT__ENABLED` | `true` | Open-access full-text grounding |
+| `RA_FULLTEXT__MAX_PAPERS` | `5` | Top papers attempted per run |
+| `RA_FULLTEXT__MAX_PDF_MB` | `15` | Per-PDF download size cap |
+| `RA_FULLTEXT__TOP_CHUNKS_PER_PAPER` | `3` | Passages retrieved per paper |
+
+Full key lists live on the stage pages: [Snowball](../architecture/stages/snowball.md), [Rerank](../architecture/stages/rerank.md), [Fulltext](../architecture/stages/fulltext.md).
 
 ---
 

mkdocs.yml

@@ -79,6 +79,7 @@ nav:
           - architecture/stages/snowball.md
           - architecture/stages/rerank.md
           - architecture/stages/relevance-scoring.md
+          - architecture/stages/fulltext.md
           - architecture/stages/clustering.md
           - architecture/stages/synthesis.md
           - architecture/stages/gap-analysis.md