Chat with your own documents and get grounded, cited answers — not hallucinations.
docchat-rag is a small, production-minded Retrieval-Augmented Generation (RAG)
application. You upload PDF / TXT / Markdown files, it ingests them with
boundary-aware semantic chunking, stores embeddings in a persistent
ChromaDB, and answers your questions strictly
from the retrieved context — always showing which document and which
page/section each answer came from. If the answer isn't in your documents, it
says so instead of making something up.
It runs 100% locally out of the box (Ollama + local sentence-transformers embeddings, no API key, nothing leaves your machine), with cloud API mode (OpenAI / Anthropic) available as an opt-in.
This repository is a focused engineering sample. It shows:
- Grounded generation with citations — every answer is traceable to a source chunk (document + page/section), the core trust feature of a usable RAG system.
- An anti-hallucination guardrail — when the retrieved context does not contain the answer, the app says so instead of inventing one.
- Hybrid retrieval — dense vector search fused with lexical BM25 via Reciprocal Rank Fusion, with an optional cross-encoder re-ranking stage, so both meaning and exact-term queries are served well (see Retrieval).
- Boundary-aware semantic chunking — not naive fixed-size splitting. Text is split along document structure first, then optionally refined with an embedding-similarity semantic breakpoint pass (see Chunking).
- A provider-agnostic architecture — LLM and embedding backends are
switchable via
.envbetween local (Ollama / sentence-transformers) and cloud (OpenAI / Anthropic) without touching application code. - Privacy / on-prem capability — the default path requires no third-party API and keeps documents and queries entirely local.
- Clean, modular, typed code — ingestion, retrieval, vector store, LLM and UI are separated and independently testable.
To record this demo, see the Demo Recording Checklist.
┌──────────────┐
PDF/TXT/MD ──► │ Ingestion │ loaders → semantic chunking → embeddings
└──────┬───────┘
▼
┌──────────────┐
│ ChromaDB │ persistent vector store (chunks + metadata)
└──────┬───────┘
▼
question ──► ┌──────────────┐ retrieve top-k chunks
│ Retrieval │ ───────────────────────┐
└──────────────┘ ▼
┌──────────────┐
│ LLM │ grounded
│ (Ollama/API) │ answer +
└──────┬───────┘ citations
▼
Streamlit chat UI
Module map:
| Path | Responsibility |
|---|---|
src/ingestion/loaders.py |
Extract text from PDF / TXT / MD (with page numbers for PDFs). |
src/ingestion/chunking.py |
Structure-first + semantic-breakpoint chunking. |
src/ingestion/pipeline.py |
Orchestrate extract → chunk → embed → store. |
src/embeddings/providers.py |
Switchable embedding backends (local / OpenAI). |
src/vectorstore/chroma_store.py |
Persistent ChromaDB wrapper (citation metadata preserved). |
src/retrieval/retriever.py |
Query the store (vector or hybrid), optional re-rank, return chunks + metadata. |
src/retrieval/bm25.py |
Dependency-free BM25 (Okapi) lexical index for the hybrid pass. |
src/retrieval/fusion.py |
Reciprocal Rank Fusion to combine dense + lexical rankings. |
src/retrieval/reranker.py |
Optional injectable cross-encoder re-ranker (lazy sentence-transformers). |
src/llm/providers.py |
Switchable LLM backends (Ollama / OpenAI / Anthropic). |
src/llm/prompt.py |
Grounded system prompt, context formatting, citation parsing, guardrail. |
src/rag.py |
RAG engine: retrieve → ground → generate → cite. |
src/eval/ |
Offline-first deterministic evaluation harness (metrics, datasets). |
app.py |
Streamlit chat UI (upload, ingest, chat, citations). |
scripts/ingest.py |
CLI: ingest files/folders into the store. |
scripts/evaluate.py |
CLI: run the evaluation harness against the gold dataset. |
scripts/inspect_store.py |
CLI: inspect stored chunks + metadata. |
scripts/ask.py |
CLI: ask a grounded question and see its citations. |
| Concern | Default (local, free) | Opt-in (cloud) |
|---|---|---|
| LLM | Ollama llama3.2 (multilingual alt: qwen2.5:1.5b-instruct) |
OpenAI gpt-4o-mini or Anthropic Claude |
| Embeddings | BAAI/bge-small-en-v1.5 (sentence-transformers) |
OpenAI text-embedding-3-small |
| Vector store | ChromaDB (persistent, on disk) | same |
The default local LLM is chosen to be lightweight and fast (3B parameters), easily fitting in VRAM and leaving ample headroom for context.
Note: Anthropic does not offer an embeddings API, so in cloud mode embeddings run on OpenAI or stay local. LLM and embedding providers are configured independently for exactly this reason.
See the implementation in src/ingestion/chunking.py.
- Structure-first split. The document is broken into structural blocks
(Markdown headings, paragraphs, list items, fenced code blocks) so a chunk
never starts or ends mid-structure. Heading context is tracked and attached to
each chunk as a
sectionfor citations. - Sentence-aware packing. Blocks are split into sentences, then greedily packed into chunks up to a token budget with a configurable overlap — chunks never cut a sentence in half.
- Semantic breakpoints (optional). When an embedding function is supplied, consecutive-sentence cosine similarity is computed and a breakpoint is inserted where similarity drops below a percentile threshold, so chunks align with topic shifts rather than arbitrary length.
The chunker has no hard dependency on any ML library — the embedding function
is injected — so it is fully unit-testable offline (see tests/test_chunking.py).
See src/retrieval/. Retrieval runs in one of two modes,
selected by RETRIEVAL_MODE:
vector— embed the question and pull the nearest chunks from ChromaDB. Strong on meaning and paraphrase, weak on exact terms.hybrid(default) — additionally run a lexical BM25 pass over the corpus and fuse the two rankings with Reciprocal Rank Fusion (RRF). This recovers exact-term hits (names, codes, acronyms, rare tokens) that dense search alone tends to bury, with no extra dependency or model download.
Optionally, either mode can be followed by a cross-encoder re-ranker
(RERANK_ENABLED=true): a generous candidate pool (RETRIEVAL_FETCH_K) is
fetched cheaply, then re-scored jointly per (question, chunk) pair — much more
precise than independent scoring — and the best RETRIEVAL_TOP_K are kept. The
re-ranker is injected (a small PairScorer protocol), so the fusion-and-rank
logic is fully unit-testable offline with no model. It's off by default because
it downloads a small extra model on first use.
The lexical index (bm25.py) and fusion (fusion.py) are pure Python with no
ML dependency — the same design choice as the chunker — so the trust-critical
ordering logic is covered by offline tests in tests/test_retrieval.py.
See src/llm/prompt.py and src/rag.py.
- The retriever embeds the question and pulls the top-k chunks from ChromaDB, each still carrying its citation metadata (document, heading path, chunk index, page).
- Those chunks are rendered into a numbered context block (
[1] Source: …) and passed to the LLM with a strict system prompt: answer only from the context, cite the excerpts you used with their bracketed numbers, treat the context as untrusted data, and if the answer isn't present reply with one exact sentence. - The RAG engine reads the model's reply: it maps the inline
[n]markers back to the source chunks (showing the same numbers in the UI), and it detects the "not in the documents" sentence to flag the answer as not grounded — the anti-hallucination guardrail. When the guardrail fires, no sources are surfaced because none support an answer.
This logic is covered by offline tests with mock LLMs in tests/test_rag.py.
docchat-rag includes an offline-first, deterministic evaluation harness to measure the trust-critical properties of the system without relying on an LLM-as-a-judge.
The harness evaluates four strict metrics against a hand-crafted Gold Dataset (src/eval/dataset.py) based on the bundled sample document:
- Retrieval hit rate: Did the retrieval system surface a relevant chunk?
- Answer accuracy: Did the LLM synthesize the correct factual response based on minimal core keywords?
- Citation accuracy: Do the surfaced citations genuinely support the answer?
- Guardrail accuracy: Are unanswerable questions correctly refused (no hallucinations), and answerable questions answered (no false refusals)?
All metrics are pure functions (in src/eval/metrics.py) and fully unit-testable.
To run the evaluation:
# Ensure the sample document is ingested first
python scripts/ingest.py
# Run the evaluation harness
python scripts/evaluate.pyRequires Python 3.12. Tested on Windows 11.
# 1. clone & enter
git clone <your-fork-url> docchat-rag
cd docchat-rag
# 2. create a virtual environment
python -m venv .venv
# Windows:
.venv\Scripts\activate
# macOS/Linux:
# source .venv/bin/activate
# 3. install dependencies
pip install -r requirements.txt
# 4. configure (defaults to 100% local)
copy .env.example .env # Windows
# cp .env.example .env # macOS/LinuxFor the default local setup, install Ollama and pull the model:
ollama pull llama3.2streamlit run app.pyThen, in the browser: upload PDF/TXT/MD files in the sidebar and click Ingest
(or pre-load the sample with python scripts/ingest.py), then ask questions in
the chat. Each answer shows its sources; questions not covered by your documents
return the "not in the documents" response.
python scripts/ingest.py # ingest the bundled sample
python scripts/ingest.py path/to/docs --reset # ingest your own files
python scripts/inspect_store.py --limit 5 # inspect stored chunks + metadata
python scripts/ask.py "How much parental leave do new parents get?"
python scripts/ask.py "What is the company's stock price?" # -> guardrailpip install -r requirements-dev.txt
pytest -qdocchat-rag includes a Dockerfile and docker-compose.yml for easy containerization.
# Build and run the container in detached mode
docker-compose up -dThe Streamlit UI will be available at http://localhost:8501.
Note on Ollama: The app needs to communicate with Ollama. If you are running Ollama on your host machine, edit your .env to point to the host:
OLLAMA_BASE_URL=http://host.docker.internal:11434
Alternatively, you can uncomment the ollama service block in docker-compose.yml to run it as a container.
Note on persistence: The chroma_db directory is mounted as a named volume, ensuring your ingested documents persist across container restarts. Embedding models will download on the first run.
The repository includes a GitHub Actions pipeline (.github/workflows/ci.yml) that runs fully offline tests and linting on every push and pull request.
A Makefile is included for common tasks:
make install(installs dependencies)make lint(runsruff)make test(runspytest)make run,make ingest,make eval,make docker-build,make docker-run
data/sample_docs/acme_employee_handbook.md is a fictional company handbook
included so the demo works immediately with no real or proprietary content.
- Scanned / image-only PDFs have no extractable text; OCR is out of scope. Such files are reported as "no extractable text" rather than failing silently.
- Prompt-injection from document content is mitigated (the system prompt instructs the model to treat context as data) but, as with any LLM, not perfectly eliminated.
- Very long questions or chunks are truncated by the embedding model's input limit (512 tokens for bge-small).