DocuMind API

Cited document Q&A with a published retrieval benchmark.

Upload documents, ask questions, get answers with citations you can click back to the source — and a measured comparison of which retrieval strategy actually works, and what it costs.

Status: deployed and working; the benchmark has not been run. Every results table below is empty on purpose — filling them in with invented figures would defeat the entire point of the project. Reproduce them: about twenty minutes and a few dollars of API credit.

Verified in production (Vercel sin1 + Neon Postgres 18 / pgvector 0.8.1): a PDF ingests, hybrid retrieval returns ranked passages, the answer carries citations that resolve and verify at 100% coverage, cost and latency are reported per request, an unanswerable question is refused, and the SSE stream delivers tokens progressively rather than in one flush. pytest: 108 passing.

Not yet exercised: the eval harness and benchmark table, the HNSW sweep, the Qdrant backend, Langfuse tracing, and the Kubernetes manifests. All are written; none has been run.

Live API: https://documind-api-theta.vercel.app (health · docs) · Frontend: documind-web

---

Why this exists

Every AI portfolio has a RAG chatbot. Almost none has an evaluation section with numbers in it.

So this project is built around the measurement rather than the demo:

• A 30-question golden set with a deliberate category mix — including five questions whose correct answer is "I don't know", and four that hinge on an exact term a dense embedding smooths away.
• Five configurations, each changing exactly one variable from the previous one, so a difference in the table is attributable to that change and nothing else.
• A runtime answer-quality layer — per-claim citation verification, a composite confidence score, and a structured abstention path — because an average hallucination rate measured offline does not catch the specific hallucination in front of a user right now.
• An HNSW recall/latency sweep measured against exact search, not against another approximation.

---

Architecture

                     ┌──────────────────────────────────────────────┐
  POST /documents ──▶│  ingest (resumable state machine)            │
  POST  ../ingest ──▶│  parse ─▶ chunk ─▶ embed ─▶ dedupe ─▶ index  │──▶ Postgres + pgvector
      (poll)         │    │                                          │    (or Qdrant)
                     │    └─ low-text page? ─▶ vision model          │
                     └──────────────────────────────────────────────┘

                     ┌──────────────────────────────────────────────┐
  POST /query/stream │  guardrails ─▶ rewrite ─▶ retrieve            │
                 ──▶ │    vector ─┐                                  │
   SSE events ◀───── │    lexical ─┴─ RRF ─▶ rerank (50 ─▶ 5)        │
   meta              │      │                                        │
   retrieval         │      ├─ confidence too low? ─▶ abstain        │
   token…            │      └─▶ generate ─▶ verify claims ─▶ score   │
   verification      └──────────────────────────────────────────────┘
   confidence                                │
   usage                                     └──▶ Langfuse (optional)
   done

  GET /evals/benchmark ──▶ the table below, from stored eval runs

The three decisions worth explaining

Ingestion is a resumable state machine, not a queue. The deployment target is Vercel, which has no background workers — anything started after the response is written dies when the sandbox freezes. So progress lives in Postgres: each POST /documents/{id}/ingest advances the pipeline as far as a wall-clock budget allows, commits, and returns. The client polls. This is crash-safe (a tick that dies loses at most one batch) and runs identically under docker compose, where a driver loop replaces the poller. The cost is chattiness, which is a fair trade against operating a queue.

Embeddings live in per-space physical tables. pgvector needs a fixed dimension on the column to build an HNSW index, but config E deliberately compares a 384-dim local model against config D's 1536-dim API model. Each ingestion space — a fingerprint of chunk strategy, size, overlap, and embedder — gets its own chunk_emb_* table. This is also what stops config A from silently retrieving config B's chunks and turning the benchmark into a comparison of a config against itself.

No database session is held during generation. Retrieval finishes and releases its connection before the first token is requested. Holding a Postgres connection for the 5–20 seconds a model takes to stream is the easiest way to exhaust a serverless connection pool, and it buys nothing.

---

Evaluation

The configurations

#	Config	Isolates
A	Fixed-size chunks, pure vector, top-5	baseline
B	Recursive + 15% overlap, pure vector, top-5	does chunking strategy matter?
C	B + hybrid (lexical + vector, RRF)	does keyword search rescue exact-term queries?
D	C + rerank (retrieve 50, keep 5)	is reranking worth the latency?
E	D with a local bge-small embedder	should you self-host embeddings?

Results

Config	Recall@5	MRR	Faithfulness	Correctness	Refusal acc.	p95 latency	$/query
A — fixed, vector
B — recursive + overlap
C — + hybrid RRF
D — + rerank
E — D w/ local embeddings

Interpretation — write 3–5 sentences here once you have run it. The numbers are the evidence; this paragraph is the engineering. Say which change produced the largest gain and in which category it was concentrated, name the trade you would accept and the one you would not, and be specific about what you would do differently for a latency-sensitive deployment.

Category breakdown

The aggregate table hides the interesting part. Configs mostly agree on plain factual lookups and diverge sharply on the exact-term and unanswerable buckets — which is exactly why those buckets are in the golden set.

Category	n	What it tests
factual	10	baseline retrieval and grounding
multi-hop	8	needs 2+ chunks, often across documents
unanswerable	5	the correct answer is a refusal
exact-term	4	acronyms and identifiers, where dense retrieval loses to lexical
ambiguous	3	underspecified — a good answer names the ambiguity

Refusal is reported three ways, on purpose

A system that refuses everything scores perfectly on "did it refuse the unanswerable ones". So the harness reports refusal_recall (of the unanswerable, how many refused) alongside false_refusal_rate (of the answerable, how many refused anyway). Quoting the first without the second is the most common way a RAG benchmark flatters itself.

Vector index: recall vs latency

index	ef_search	recall	p50	p95
exact	—	1.000
hnsw	10
hnsw	40
hnsw	80
hnsw	320

Exact search runs first to establish ground truth. Without it you are comparing one approximation against another, which measures agreement rather than recall — and that is the step most ANN benchmarks skip. Pick the operating point from the knee and state the recall you traded for the latency you gained.

---

The answer-quality layer

Retrieval quality is what the benchmark measures. This is the layer that decides what the user is allowed to see, and it is the part most RAG projects skip.

Citation verification

Resolving a citation is not verifying it. [C3] pointing at a real chunk only proves the model wrote a label that exists. A model will cite a genuine passage for a claim that passage never makes, and it looks perfect in the UI.

So there are two steps: resolution (free — does the label map to a passage we actually retrieved?) and verification ($ — one judge call per claim/citation pair asking whether the passage entails the claim). Unsupported claims are surfaced in the UI by default and can be gated instead via config.

Faithfulness measured on an eval set tells you your average hallucination rate. Per-request citation verification catches the specific hallucination in front of the user. They are different tools and this ships both.

Composite confidence

Three signals, none of which costs an extra model call, reported as a breakdown rather than one number:

Signal	Weight	Source
Retrieval confidence	0.35	reranker scores of the chunks actually cited
Citation coverage	0.45	% of claims with a verified citation
Completeness	0.20	did the answer address every part of the question?

Grounding outweighs retrieval deliberately: a well-retrieved answer that cites nothing is worse than a thinly-retrieved answer that cites correctly. "87% confident" is unactionable; "retrieval strong, two claims uncited" tells the reader which part to distrust.

Structured abstention

When retrieval confidence falls below threshold, nothing is generated. Instead:

{
  "answered": false,
  "found": "3 passages about Q3 revenue recognition policy",
  "missing": "no passage covering FY2024 segment-level revenue",
  "suggested_sources": [{ "doc": "10-K_2024.pdf", "pages": [41, 42], "reason": "segment tables" }]
}

The five unanswerable questions exist to measure that this fires when it should — and, just as importantly, that it does not fire on answerable ones.

---

Running locally

cp .env.example .env          # add OPENAI_API_KEY
docker compose up --build
open http://localhost:8000/docs

That brings up Postgres with pgvector, runs migrations, and starts the API with the local cross-encoder and embedding model available (configs D and E need them).

Without Docker:

uv venv && uv pip install -r requirements.txt -r requirements-local.txt
alembic upgrade head
uvicorn app.main:app --reload

Already running Postgres on 5432? Set POSTGRES_PORT=5433 in .env and update DATABASE_URL to match — compose reads both.

Tests

pytest -q                          # 76 tests, no database or API key required
ruff check app scripts tests
python scripts/smoke_test.py       # end-to-end against a real database
python scripts/smoke_test.py --no-llm   # schema and SQL only, no API cost

The smoke test builds its own PDF with known content, ingests it, and asserts the whole chain: migrations applied, enum labels lowercase, the tsvector GIN index present, embeddings written to the per-space table, lexical search finding a literal identifier that dense retrieval smooths away, citations resolving, and a refusal on a question the corpus cannot answer. That is the ground unit tests cannot cover.

The suite covers chunk metadata, RRF, the grading arithmetic, cost accounting, the guardrail false-positive cases, and an ORM↔migration contract check. That last one exists because a real bug shipped without it: SQLAlchemy persists Python enums by .name, so DocumentStatus.UPLOADED was being written as "UPLOADED" into a Postgres enum with lowercase labels. Every insert would have failed at runtime, and no unit test touched a database.

---

Reproducing the benchmark

python scripts/fetch_corpus.py              # 13 open-access arXiv papers on retrieval and RAG
python scripts/ingest_corpus.py --all       # 3 distinct ingestion spaces across the 5 configs
python scripts/verify_golden_set.py         # ⚠️ do not skip — see below

Verify the golden set before trusting any number. The shipped expected_chunks anchors were authored against the published content of those papers but have not been machine-checked against your extracted copies. An anchor that matches nothing is invisible during a run: recall just comes out lower, identically for every config, and the whole table looks like a hard corpus rather than a broken ground truth. The verifier reports every unresolvable anchor so you can fix it.

Then:

for c in A B C D E; do python -m app.evals.run --config $c; done
python -m app.evals.sweep --output evals/out/sweep.json
python -m app.evals.compare --table --markdown   # paste into the table above

Anchors come in two forms. paper.pdf#p12 is a page anchor; paper.pdf#~late interaction is a content anchor, matching any chunk from that document containing the phrase. Content anchors are preferred here because they survive re-chunking — a page anchor silently changes meaning when chunk size changes, which would corrupt a benchmark whose entire purpose is comparing chunk sizes.

---

LLMOps

Prompt versioning. Prompts are files in git named <name>.v<n>.md, loaded by version. The resolved ID is logged on every request and stored on every eval run, so a quality change six weeks later is traceable to a specific prompt. Once numbers have been published against a version, you add .v2.md rather than editing in place.

Eval regression gate. .github/workflows/evals.yml runs the golden set on any PR touching prompts, retrieval, or the quality layer, and fails if a gated metric drops more than 3 points against main. Cost and latency are reported but deliberately not gated — they are a trade, not a failure. A missing baseline passes with a warning, because a gate that fires when it has no information teaches people to bypass it.

Tracing. Langfuse, with one trace per query and spans for retrieval, rerank, and generation. Entirely optional — absent keys make it a no-op, and a tracing failure can never take down a request.

Model pinning. Every model ID is an explicit version, never a floating alias. A silent model swap is a silent quality regression that the eval gate cannot attribute.

Guardrails. Injection heuristics and PII detection on input; citation resolution and groundedness checks on output. The degradation ladder is: timeout → bounded retry → secondary model → return the retrieved passages without generation → honest error. Answering from the model's own knowledge is not on the ladder — a silently ungrounded answer is the only failure the user cannot detect.

Kubernetes. k8s/ — Deployment, Service, ConfigMap, Secret, HPA, PDB, verified on k3d. See k8s/README.md for why liveness and readiness point at different endpoints.

---

Deploying to Vercel

Import the repo at vercel.com/new. Project settings:

Setting	Value
Application Preset	FastAPI
Root Directory	./
Install Command	leave as the default
Build / Output	leave empty

Vercel resolves the entrypoint by scanning app.py/index.py/main.py/server.py/wsgi.py/ asgi.py at the root and inside src/, app/ and api/, then bundles the whole app as a single function. app/main.py is the only file in this repo exposing a module-level app, so resolution is unambiguous, and vercel.json configures that same path.

Two packaging details that are easy to get wrong, both enforced by tests/test_deployment_contract.py:

• pyproject.toml is excluded from the upload (.vercelignore). Vercel runs uv lock against it whenever it is present and never reads requirements.txt. Since this repo's pyproject.toml holds only ruff and pytest config, leaving it in breaks the build.
• .vercelignore patterns are anchored with /. It uses gitignore semantics, so an unanchored evals/ also matches app/evals/ and strips a package the request path imports.

Environment variables:

DATABASE_URL     postgresql://…-pooler….neon.tech/db?sslmode=require   # paste verbatim
OPENAI_API_KEY   sk-…
ACTIVE_CONFIG    serverless
ENVIRONMENT      vercel
CORS_ORIGINS     https://<your-frontend>.vercel.app

DATABASE_URL is normalised on load, so the console's connection string works as-is — the driver is corrected and sslmode/channel_binding are translated into asyncpg connect arguments. Use the pooled host; the direct one exhausts connections under serverless.

Run alembic upgrade head locally against that database first. The app does not migrate on boot, because two concurrent instances racing the same migration is worse than a manual step.

What is different on serverless, stated plainly

	Container	Vercel
Reranker	real cross-encoder (bge-reranker-base)	LLM scorer — torch does not fit in a 250 MB bundle
Local embeddings (config E)	available	not available
Ingestion	driver loop, 120s ticks	client polls, 40s ticks
Rate limiting	in-process	Postgres-backed (per-process counters enforce nothing across invocations)
OCR	vision model	vision model (no tesseract binary either way)

ACTIVE_CONFIG=serverless is a distinct config, not config D, so the deployment never reports D's numbers for a reranker it is not running.

Streaming works. Measured against the live deployment, 33 token events arrived over a 2.23 s spread rather than in a single flush, so the progressive reveal is real and not a local-only behaviour. (The client parser handles a buffered response identically, should that ever change.)

Latency is the honest weak point: ~10 s to first token on a cold function, of which ~8 s is retrieval — cold start, query rewrite, embedding, hybrid search, and the LLM reranker, each a round trip to ap-southeast-1. Warm requests are far quicker. Cutting it means dropping the rewrite and rerank hops, which is exactly the trade the benchmark exists to quantify.

---

Limitations

Honest limitations read as senior; overclaiming is the fastest way to lose a technical reviewer.

• 30 questions over 13 documents is a demo-scale benchmark, not a production one. The confidence interval on a 30-question set is wide enough that small differences between configs are noise. Treat gaps under ~5 points as unresolved.
• The judge shares a model family with the generator, which is measurably more generous. It is tolerable for comparing configs, since the bias is constant across the comparison, and not tolerable for an absolute claim like "94% faithful".
• "BM25" is really Postgres ts_rank_cd — a cover-density rank with no explicit document length normalisation. It behaves like BM25 for the purpose that matters here (matching literal tokens dense embeddings smooth away), but it is not the textbook algorithm.
• No multi-tenancy. tenant_id exists on every row and is always "default". The column is there so adding it is not a re-index; the auth and isolation work is not done.
• Reranking latency is untested above a handful of concurrent users, and the HPA scales on CPU, which is a poor proxy for a workload that spends most of its time waiting on an upstream API.
• PDF-only ingestion. No HTML, DOCX, or plain text.
• Injection defence is heuristic. The structural control — the model can only answer from supplied passages, and the answer is verified against them — matters far more than the regexes.

What I would do next

Real multi-tenancy with row-level security · a requests-per-second custom metric for the HPA · a proper BM25 implementation to replace ts_rank_cd · a larger golden set with inter-annotator agreement · caching at the semantic level, measured rather than assumed.

License

MIT