Roadmap P2: full-text grounding — open-access RAG with verbatim evidence

[Ndevu12]

Summary

Implements Phase 2 of the improvement roadmap — the largest single quality jump it identifies. The pipeline no longer reasons over abstracts alone: for the top relevance-filtered papers it resolves open-access PDFs, parses and chunks the full text, retrieves the query-relevant passages, and grounds synthesis in them. Reports now show verbatim evidence quotes under each paper, making claims checkable against sources.

The src/fulltext package (previously interface stubs)

• Resolver — metadata-first, zero extra API calls: arXiv IDs map straight to PDF URLs, OpenAlex work records already carry open-access locations in their raw search metadata, CORE results expose download URLs. Closed-access papers resolve to nothing.
• CachingPDFDownloader — PDFs cached under data/fulltext/ keyed by paper ID, with size (15 MB default) and content-type guards.
• Parser — PyMuPDF extraction with heuristic section detection (Abstract, Introduction, Methods, …); the references section is truncated so bibliography lines don't pollute retrieval.
• SectionAwareChunker — overlapping ~1400-char windows, each tagged with the section it came from.
• InMemoryFulltextIndex — chunks embedded through the existing embedding provider (reusing its disk cache) and searched by cosine similarity, with BM25 fallback when the embedding backend is unavailable — grounding works in every environment.

The fulltext pipeline stage

Runs between relevance scoring and clustering, budgeted by a new fulltext config section (papers attempted, PDF size cap, timeouts, chunk sizing, passages per paper). Query-relevant passages per paper land in the fulltext_passages artifact. Best-effort throughout: closed-access papers, failed downloads, malformed PDFs, or a missing pymupdf reduce coverage but never break a run.

Evidence in synthesis and reports

• PaperExtraction and PaperAnalysis gain an evidence field.
• LLM extraction prompts embed the retrieved passages and require short verbatim quotes (system prompt and schema updated); heuristic extraction attaches trimmed passages directly, and its provenance note now says "abstract and full-text passages".
• The synthesis stage and its timeout-recovery path both read the passages artifact.
• Markdown reports render the quotes under each paper as Evidence (from full text) blockquotes.

Environment variable documentation

.env.example and the authoritative environment-variables reference now document everything the app reads: the new provider keys (NCBI_API_KEY, CORE_API_KEY), provider enable toggles, console/setup controls (RA_CONSOLE_LOG_LEVEL, RA_SKIP_SETUP_CHECK), and the accuracy-stage namespaces (RA_SNOWBALL__*, RA_RERANK__*, RA_FULLTEXT__*). Two hygiene fixes: RA_DEBUG=1 is no longer active in the example (copying it previously enabled debug mode on every new install), and the stale Anthropic model example was replaced with a current model ID.

Dependency

pymupdf (1.28.2) added to Pipfile and Pipfile.lock. The lock was updated with the package's PyPI hashes and re-synced via pipenv's own Pipfile-hash computation, and pipenv verify confirms it is up to date (full pipenv lock could not run in this environment because the pytorch-cpu index in the Pipfile sources is unreachable from it).

Verification

• pipenv run pytest tests/ — 417 passed (19 new fulltext tests: resolver, a real PyMuPDF parse of a generated PDF with section detection and references truncation, chunker, BM25-fallback retrieval scoped per paper, stage artifact production, graceful-degradation paths, evidence in prompts/heuristics/rendering), 1 skipped — re-verified after merging main (Roadmap P1: implement PubMed, DBLP, and CORE providers #35)
• ruff check src tests setups — clean
• scripts/check_docs_policy.py — passes (61 nav pages)
• pipenv verify — Pipfile.lock in sync
• Live PDF downloads could not be exercised here (scholarly hosts blocked by this environment's proxy); the download path is covered by fakes and guards — recommend one live run against an arXiv-heavy query after merge