A self-hosted, domain-specific Retrieval-Augmented Generation (RAG) knowledge assistant. Upload your own documents (PDF, DOCX, Markdown, CSV, JSON, HTML, TXT), and ask questions in natural language. Answers are generated by a local language model, grounded in the retrieved chunks from your documents, and returned with citations back to the exact source passages — everything runs on your own infrastructure, with no third-party API calls.
| Backend | FastAPI (Python 3.11) · SQLAlchemy 2.0 · PostgreSQL · Redis · Celery · ChromaDB · Hugging Face Transformers (Qwen2.5-0.5B-Instruct) · LangChain |
| Frontend | Next.js 16 (App Router) · React 19 · Tailwind CSS v4 |
| Delivery | JSON REST API + Server-Sent Events (SSE) token streaming |
- Key features
- High-level architecture
- Repository layout
- Technology stack
- Backend documentation
- Frontend documentation
- Getting started
- Development & testing
- Deployment
- License
- Async ingestion pipeline — uploads are validated, stored and dispatched to a Celery worker (or run inline) that parses, chunks, embeds and indexes the document, tracking progress in a durable task ledger.
- Embedded, persistent vector search — ChromaDB runs inside the Python process and persists to disk; no separate search service to operate.
- Per-user isolation — every vector carries its owner's
user_id; retrieval injects a Chromawhereclause so a user can never see another tenant's chunks. - Local LLM inference — grounded answers from Qwen2.5-0.5B-Instruct served
through
transformers, plus a deterministicstubprovider for tests/CI. - Real token streaming — answers arrive over SSE as the model produces them
(
start → sources → token* → done), rendered live in the console. - Citations — each answer carries the retrieved source chunks (with snippet and similarity score) that grounded it.
- JWT auth with refresh — access/refresh token pair, transparent client-side refresh, scope-aware verification.
- Manual, scope-aware rate limiting — a hand-written sliding-window-log
limiter (Redis-backed, in-memory fallback) protecting
auth,chat,uploadand general traffic. - Uniform error envelope — every failure returns the same JSON shape
(
code,message,details,request_id). - Observability — request-id correlation, structured access logs,
X-Request-ID,X-Process-Time-MsandX-RateLimit-*response headers, plus liveness/readiness probes that check every dependency. - Dark/light/system theming and a responsive, dependency-light UI.
┌──────────────────────────────────────────────────────────────────────────┐
│ Browser │
│ Next.js 16 App Router (React 19) · Tailwind CSS v4 │
│ ┌──────────────┐ ┌───────────────┐ ┌───────────────┐ ┌────────────┐ │
│ │ Chat │ │ Documents │ │ Dashboard │ │ Settings │ │
│ └──────────────┘ └───────────────┘ └───────────────┘ └────────────┘ │
└───────────────────────────────┬──────────────────────────────────────────┘
fetch + Bearer JWT │ SSE (fetch + ReadableStream)
▼
┌──────────────────────────────────────────────────────────────────────────┐
│ FastAPI application (backend/) │
│ Middleware: RequestContext → RateLimitHeaders → Security → GZip → CORS │
│ Routers: /api/v1/{health, auth, users, documents, chat, tasks} │
│ │
│ Services ── DocumentService · ChatService · AuthService · TaskService │
│ RAG ── RAGChain · Retriever · PromptBuilder · Ingestion Pipeline │
│ LLM ── TransformersLLMProvider (Qwen2.5-0.5B) · StubLLMProvider │
│ Stores ── SQLAlchemy (PostgreSQL) · ChromaDB (embedded) · Redis cache │
└───────┬─────────────────────┬───────────────────────┬─────────────────────┘
│ │ │
▼ ▼ ▼
PostgreSQL Redis ChromaDB (on-disk)
users, documents, cache, rate-limit knowledge_base collection
chunks, conversations, counters, Celery (vectors + metadata)
messages, tasks broker + backend
▲
│ consume "ingestion" / "maintenance" queues
┌───────┴──────────────────────────────────────────────────────────────────┐
│ Celery worker (same image) │
│ ingest_document · reindex_document · cleanup_stale_tasks · health │
└──────────────────────────────────────────────────────────────────────────┘
Design principles
- Layered backend — HTTP (
api/) → business logic (services/) → persistence (repositories/+models/) → infrastructure (db/,workers/,rag/,llm/). Services own transactions; repositories never commit. - Provider abstraction — both the LLM (
LLMProvider) and embeddings (EmbeddingProvider) are interfaces with a production implementation and a deterministic, dependency-free one for tests, selected by configuration. - Lazy singletons — the vector store, retriever, embedding provider, LLM provider, cache, rate limiter and ingestion pipeline are built once per process behind a lock, so heavy models load at most once.
- Client is a client-rendered SPA shell — tokens live in
localStorage; the API is the single source of truth for authorisation.
v1/
├── README.md # This file
├── backend/ # FastAPI service + Celery worker
│ ├── Dockerfile # Multi-stage image (API + worker share it)
│ ├── requirements.txt # Python dependencies (range-pinned)
│ ├── .env # Committed backend configuration (source of truth)
│ ├── app/
│ │ ├── main.py # Application factory + lifespan + middleware
│ │ ├── api/ # HTTP layer (routers, deps, middleware, errors)
│ │ ├── core/ # Config, security, logging, rate limiting, exceptions
│ │ ├── db/ # Engine/session, cache, vector store, init
│ │ ├── llm/ # LLMProvider interface, transformers + stub impls, prompts
│ │ ├── models/ # SQLAlchemy ORM models
│ │ ├── rag/ # Loaders, chunking, embeddings, retriever, chain, pipeline
│ │ ├── repositories/ # Data-access layer (CRUD, no commits)
│ │ ├── schemas/ # Pydantic request/response models
│ │ ├── services/ # Business logic (auth, documents, chat, tasks)
│ │ └── workers/ # Celery app + ingestion & maintenance tasks
│ ├── tests/ # Pytest: unit/ and integration/
│ └── data/ # Runtime artefacts (chroma, uploads, hf cache)
└── frontend/ # Next.js console
├── Dockerfile # Multi-stage (standalone) image
├── package.json # next 16 · react 19 · tailwindcss v4
├── next.config.mjs # output: "standalone", turbopack, cacheComponents
├── jsconfig.json # "@/*" → "./src/*" path alias
└── src/
├── app/ # App Router routes (public + /app console)
├── components/ # ui.js, shell.js, icons.js, auth-shell.js
└── lib/ # api.js, sse.js, auth.js, theme.js, toast.js
| Concern | Choice | Notes |
|---|---|---|
| Web framework | FastAPI >=0.115,<0.116 |
Async, OpenAPI, dependency injection |
| ASGI server | Gunicorn + Uvicorn workers | UvicornWorker, 2 workers, 900s timeout |
| Validation | Pydantic v2 + pydantic-settings | Typed settings + request/response schemas |
| ORM / DB | SQLAlchemy 2.0 + psycopg2 | PostgreSQL only (no SQLite in production) |
| Migrations | Alembic (declared) | create_all() used for local/dev boot |
| Auth | PyJWT + bcrypt | HS256 JWTs, bcrypt password hashing (12 rounds) |
| Cache/queue | Redis 6.1+ + Celery 5.4 | Cache, rate limits, broker + result backend |
| Vector store | ChromaDB >=0.5.20,<0.7 |
Embedded + persistent, no server |
| RAG tooling | LangChain 0.3.x |
Recursive splitter, Document interop |
| Inference | transformers + torch (CPU) | Qwen2.5-0.5B-Instruct, full-precision fp32 |
| Embeddings | sentence-transformers | all-MiniLM-L6-v2 (384-dim) |
| Parsing | pypdf, python-docx, beautifulsoup4 | PDF/DOCX/HTML (+ CSV/JSON/text built in) |
| Testing | pytest, pytest-asyncio, pytest-cov, httpx | Hermetic (no external services) |
| Concern | Choice | Notes |
|---|---|---|
| Framework | Next.js 16 (App Router) | output: "standalone" for Docker |
| UI runtime | React 19 | Client components ("use client") |
| Styling | Tailwind CSS v4 | @import "tailwindcss", @custom-variant dark |
| Fonts | next/font (Geist, Geist Mono) |
Self-hosted via next/font/google |
| State | React Context (auth, theme, toast) | No external state library |
| Transport | fetch + manual SSE parser |
EventSource can't POST/send headers |
app/
├── main.py # create_app(), lifespan (init DB, warm LLM), middleware, OpenAPI
├── api/
│ ├── dependencies.py # get_current_user, get_optional_user, require_admin, pagination
│ ├── rate_limit_dep.py # rate_limit_default / _auth / _chat / _upload
│ ├── middleware.py # RequestContext, RateLimitHeaders, SecurityHeaders
│ ├── errors.py # Uniform exception handlers
│ └── v1/
│ ├── router.py # Mounts every endpoint module under /api/v1
│ └── endpoints/ # health.py, auth.py, users.py, documents.py, chat.py, tasks.py
├── core/
│ ├── config.py # Settings (pydantic-settings) + LLM budget constants
│ ├── security.py # bcrypt hashing, JWT issue/verify
│ ├── rate_limit.py # Sliding-window limiter (Redis + in-memory stores)
│ ├── logging.py # configure_logging, RequestContextFilter, get_logger
│ └── exceptions.py # AppError hierarchy (HTTP status + stable code)
├── db/
│ ├── base.py # Declarative Base + UUID/Timestamp mixins
│ ├── session.py # Engine/session factory, get_db, session_scope, configure_engine
│ ├── cache.py # CacheClient (Redis) + InMemoryCache fallback
│ ├── vector_store.py # VectorStore over Chroma (telemetry disabled)
│ └── init_db.py # create_all(), seed_default_admin(), init_database()
├── llm/
│ ├── base.py # LLMProvider ABC, Message, GenerationResult
│ ├── factory.py # build/get/reset_llm_provider (singleton)
│ ├── transformers_llm.py # Qwen2.5-0.5B inference (+ streaming)
│ ├── stub.py # Deterministic extractive provider (tests)
│ └── prompt.py # PromptBuilder (system + context + history)
├── models/ # user.py, document.py, conversation.py, task.py
├── rag/
│ ├── loaders.py # DocumentLoader (bytes → text + metadata)
│ ├── chunking.py # TextChunker (LangChain recursive splitter)
│ ├── embeddings.py # EmbeddingProvider: huggingface + hash
│ ├── retriever.py # Retriever (query embed + Chroma search + threshold)
│ ├── chain.py # RAGChain + answer post-processing (clean_answer)
│ └── pipeline.py # IngestionPipeline (load → chunk → embed → persist)
├── repositories/ # base.py, user.py, document.py, conversation.py, task.py
├── schemas/ # common.py, auth.py, document.py, chat.py, task.py
├── services/ # auth_service.py, document_service.py, chat_service.py, task_service.py
└── workers/
├── celery_app.py # Celery app (queues, eager flag, set_default)
└── tasks/ # ingestion.py, maintenance.py
- Middleware chain (outermost first):
RequestContextMiddlewareassigns/propagatesX-Request-ID, times the request and logs the access line →RateLimitHeadersMiddlewarecopies any limiter result onto the response →SecurityHeadersMiddlewareadds hardening headers →GZipMiddleware→CORSMiddleware. - Routing — the request hits a router mounted under
/api/v1. - Dependencies — auth (
get_current_user), DB session (get_db), scope rate-limit (rate_limit_*) and pagination (pagination_params) are resolved. - Service layer — the endpoint calls a service, which drives repositories
and/or the RAG chain and owns the transaction (
session.commit()). - Response — a Pydantic schema is serialised; errors are converted by the central handlers into the uniform error envelope.
Startup (lifespan) — configure logging, ensure the schema exists
(init_database, non-fatal if unavailable), create the upload/Chroma
directories, and warm the LLM weights (transformers only, skipped in tests,
never fatal). Shutdown disposes the engine only if this process created it.
All settings live in app/core/config.py (Settings) and are read from
backend/.env (committed) with an optional repo-root .env override. List values
(CORS_ORIGINS, ALLOWED_EXTENSIONS) accept JSON arrays or comma-separated
strings.
| Variable | Default | Purpose |
|---|---|---|
APP_NAME |
Enterprise RAG Knowledge Assistant |
Display name |
APP_ENV |
development |
development/staging/production/test |
DEBUG |
false |
FastAPI debug mode |
LOG_LEVEL |
INFO |
Root log level |
API_V1_PREFIX |
/api/v1 |
API mount prefix |
CORS_ORIGINS |
["http://localhost:3000","http://127.0.0.1:3000"] |
Allowed browser origins |
SECRET_KEY |
(placeholder) | JWT signing key — change in production |
JWT_ALGORITHM |
HS256 |
JWT algorithm |
JWT_ISSUER |
rag-knowledge-assistant |
iss claim, verified on decode |
ACCESS_TOKEN_EXPIRE_MINUTES |
30 |
Access-token lifetime |
REFRESH_TOKEN_EXPIRE_DAYS |
7 |
Refresh-token lifetime |
RATE_LIMIT_ENABLED |
true |
Master switch for the limiter |
RATE_LIMIT_REQUESTS / _WINDOW_SECONDS |
60 / 60 |
Default scope |
RATE_LIMIT_AUTH_REQUESTS |
10 |
Auth scope (brute-force protection) |
RATE_LIMIT_CHAT_REQUESTS |
20 |
Chat scope (expensive path) |
RATE_LIMIT_UPLOAD_REQUESTS |
10 |
Upload scope |
DATABASE_URL |
postgresql+psycopg2://rag:rag@localhost:5432/rag |
PostgreSQL DSN |
DB_ECHO / DB_POOL_SIZE / DB_MAX_OVERFLOW / DB_POOL_PRE_PING |
false/5/10/true |
Engine tuning |
REDIS_URL |
redis://localhost:6379/0 |
Cache/broker/limits |
CHROMA_PERSIST_DIR |
./data/chroma |
On-disk vector store |
CHROMA_COLLECTION |
knowledge_base |
Default collection name |
UPLOAD_DIR |
./data/uploads |
Stored uploads |
MAX_UPLOAD_SIZE_MB |
25 |
Upload ceiling |
ALLOWED_EXTENSIONS |
.pdf,.txt,.md,.csv,.docx,.json,.html |
Accepted file types |
EMBEDDING_PROVIDER |
huggingface |
huggingface or hash |
EMBEDDING_MODEL |
sentence-transformers/all-MiniLM-L6-v2 |
Embedding model |
EMBEDDING_DIMENSION |
384 |
Vector dimension |
EMBEDDING_BATCH_SIZE / EMBEDDING_NORMALIZE |
32 / true |
Embedding tuning |
CHUNK_SIZE / CHUNK_OVERLAP |
1000 / 150 |
Splitter settings |
RETRIEVAL_TOP_K / RETRIEVAL_FETCH_K |
4 / 20 |
Context size / candidate pool |
RETRIEVAL_SCORE_THRESHOLD |
0.0 |
Min similarity (raise for semantic embeddings) |
LLM_PROVIDER |
transformers |
transformers or stub |
LLM_MODEL_ID |
Qwen/Qwen2.5-0.5B-Instruct |
Model to load |
LLM_MAX_NEW_TOKENS |
512 (env) / 32768 (ceiling) |
Generation budget |
LLM_TEMPERATURE / LLM_TOP_P |
0.0 / 0.9 |
Sampling |
LLM_REPETITION_PENALTY / LLM_NO_REPEAT_NGRAM_SIZE |
1.1 / 4 |
Anti-loop controls |
LLM_DEVICE / LLM_NUM_THREADS |
cpu / 0 (all cores) |
Device & CPU threads |
LLM_ATTN_IMPLEMENTATION |
sdpa |
sdpa / eager / flash_attention_2 |
LLM_TORCH_DTYPE |
float32 |
float32/float16/bfloat16/auto |
LLM_LOW_CPU_MEM_USAGE / LLM_TRUST_REMOTE_CODE |
true / false |
Load options |
CHAT_USE_CELERY |
false |
Route generation through Celery |
HF_HOME |
./data/hf |
Hugging Face cache dir |
CELERY_BROKER_URL / CELERY_RESULT_BACKEND |
(empty → REDIS_URL) |
Celery endpoints |
CELERY_TASK_ALWAYS_EAGER |
true |
Run tasks inline (see note below) |
CELERY_WORKER_CONCURRENCY |
2 |
Worker concurrency |
Note on eager tasks. The embedded Chroma client keeps its ANN index in memory, so a separate worker process writing to the same persist directory would be invisible to the API process (retrieval would return nothing). The default
CELERY_TASK_ALWAYS_EAGER=trueruns ingestion inside the API process so the collection has a single owner. Set it tofalseonly when the API and worker are separate containers and read-after-write visibility is acceptable to trade away.
Generation-budget constants (app/core/config.py):
LLM_CONTEXT_WINDOW_TOKENS = 32_768— hard total budget (prompt + completion).LLM_MAX_GENERATION_TOKENS = 32_768— maximum tokens the model may emit; the provider trims it at request time tocontext_window − prompt_tokens.llm_max_new_tokensis validated to stay within these ceilings at configuration time.
Base URL: /api/v1. All endpoints require a Bearer access token except
/health*, /auth/register, /auth/login, and /auth/refresh. Interactive
docs: /docs (Swagger UI) and /redoc; the raw schema is at
/openapi.json (with a BearerAuth security scheme).
Common headers
| Header | Direction | Meaning |
|---|---|---|
Authorization: Bearer <jwt> |
request | Access token |
X-Request-ID |
both | Correlation id (generated if absent, echoed back) |
X-Process-Time-Ms |
response | Server-side handling time |
X-RateLimit-Limit / -Remaining / -Reset |
response | Limiter state |
Retry-After |
response (429) | Seconds until the quota resets |
Pagination — list endpoints accept page (≥1, ≤10000) and page_size
(1–100, default 20) and return a Page<T> envelope:
{ items, total, page, page_size, total_pages, has_next }.
| Method | Path | Auth | Rate scope | Description |
|---|---|---|---|---|
GET |
/health |
no | default |
Cheap liveness probe |
GET |
/health/ready |
no | default |
Deep readiness: database, cache, broker, embeddings, LLM |
| Method | Path | Auth | Rate scope | Description |
|---|---|---|---|---|
POST |
/auth/register |
no | auth |
Create an account → 201 UserRead |
POST |
/auth/login |
no | auth |
Exchange credentials → TokenPair |
POST |
/auth/refresh |
no | auth |
Rotate a refresh token → TokenPair |
GET |
/auth/me |
yes | — | Current user profile |
POST |
/auth/logout |
yes | — | Stateless logout acknowledgement |
POST /api/v1/auth/login
Content-Type: application/json
{ "email": "you@company.com", "password": "Password123" }{
"access_token": "eyJ...",
"refresh_token": "eyJ...",
"token_type": "bearer",
"expires_in": 1800,
"user": { "id": "…", "email": "you@company.com", "role": "user", "is_active": true }
}| Method | Path | Auth | Description |
|---|---|---|---|
PATCH |
/users/me |
yes | Update display name and/or password |
GET |
/users/me/stats |
yes | { documents: {...counts}, conversations, role } |
DELETE |
/users/me |
yes | Deactivate the account (soft delete) |
| Method | Path | Auth | Rate scope | Description |
|---|---|---|---|---|
POST |
/documents/upload |
yes | upload |
Multipart upload → 202, dispatches ingestion |
GET |
/documents |
yes | — | Paginated list; filters status, search, collection, extension |
GET |
/documents/stats |
yes | — | Counts per status |
GET |
/documents/{id} |
yes | — | Fetch one document |
GET |
/documents/{id}/chunks |
yes | — | Paginated stored chunks |
GET |
/documents/{id}/status |
yes | — | Ingestion progress (for polling) |
PATCH |
/documents/{id} |
yes | — | Update metadata (name, tags, collection) |
POST |
/documents/{id}/reindex |
yes | upload |
Re-run ingestion → 202 |
DELETE |
/documents/{id} |
yes | — | Delete row + chunks + file + vectors |
Upload form fields: file (required), collection (optional), tags
(optional, comma-separated), process_now (bool, default true). The upload is
streamed in ≤1 MiB chunks so the size ceiling is enforced while reading.
| Method | Path | Auth | Rate scope | Description |
|---|---|---|---|---|
POST |
/chat/completions |
yes | chat |
Non-streaming RAG answer |
POST |
/chat/stream |
yes | chat |
Stream the answer over SSE |
POST |
/chat/stream/{conversation_id} |
yes | chat |
Continue a thread and stream |
POST |
/chat/conversations |
yes | — | Create a conversation → 201 |
GET |
/chat/conversations |
yes | — | List conversations (paginated) |
GET |
/chat/conversations/{id} |
yes | — | Conversation + full message history |
GET |
/chat/conversations/{id}/messages |
yes | — | Messages of one thread |
PATCH |
/chat/conversations/{id} |
yes | — | Rename / archive |
DELETE |
/chat/conversations/{id} |
yes | — | Delete thread + messages |
ChatCompletionRequest fields: question (1–8000 chars), conversation_id?,
document_ids? (scope retrieval), top_k? (1–20), temperature? (0–2),
max_new_tokens? (16–32768), stream (ignored by SSE routes), use_history
(default true).
| Method | Path | Auth | Description |
|---|---|---|---|
GET |
/tasks |
yes | Paginated durable task audit trail |
GET |
/tasks/{task_id} |
yes | Fetch by record id or Celery id (refreshes from Celery) |
POST |
/tasks/maintenance/health |
yes | Dispatch a worker health probe → 202 |
| Method | Path | Description |
|---|---|---|
GET |
/ |
{ name, version, docs, health } (outside the schema) |
POST /api/v1/chat/stream (and /chat/stream/{conversation_id}) return
text/event-stream. Each frame is a named event whose data is a JSON object
with a type discriminator, matching ChatStreamEvent:
event: <type>
data: <json>
Frame sequence: start → sources → token* → done (or a terminal error).
| Event | Payload | Meaning |
|---|---|---|
start |
{ conversation_id, model_name } |
Stream opened |
sources |
{ sources: SourceReference[] } |
Retrieved citations, sent before generation |
token |
{ text, index } |
One progressive answer fragment |
done |
{ message_id, conversation_id, answer, latency_ms, token_count, citations } |
Final answer (accumulated tokens) + metadata |
error |
{ message, code } |
Terminal failure (HTTP status is already 200) |
heartbeat |
{} |
Keep-alive (ignored by the client) |
SourceReference = { chunk_id, document_id, document_name, score, snippet, chunk_index, cited }.
Server notes. The stream owns its own database session. Because generation
(and the SQLAlchemy session) is synchronous and CPU-bound, the service generator
runs on a dedicated worker thread whose frames are marshalled to the event
loop through an asyncio.Queue — this keeps the event loop unblocked and every
DB call on a single thread. SSE headers disable proxy buffering
(X-Accel-Buffering: no, Cache-Control: no-cache, no-transform).
event: start
data: {"conversation_id":"…","model_name":"Qwen/Qwen2.5-0.5B-Instruct"}
event: sources
data: {"sources":[{"chunk_id":"…:0","document_id":"…","document_name":"policy.pdf","score":0.83,"snippet":"…","chunk_index":0,"cited":false}]}
event: token
data: {"text":"Retention ","index":0}
event: token
data: {"text":"is 7 years…","index":1}
event: done
data: {"message_id":"…","conversation_id":"…","answer":"Retention is 7 years…","latency_ms":4120,"token_count":58,"citations":[0]}
Ingestion (app/rag/pipeline.py, run by the Celery worker or inline):
stored file ─▶ DocumentLoader ─▶ TextChunker ─▶ EmbeddingProvider ─▶ VectorStore (Chroma)
(disk) (bytes→text) (recursive) (batched embed) (+ Postgres chunk ledger)
- Load —
DocumentLoaderdispatches on extension: PDF (pypdf), DOCX (python-docx, incl. tables), HTML (beautifulsoup4), CSV, JSON, or best-effort text decode. It normalises whitespace, extracts provenance metadata and rejects empty/no-text documents. Page numbers are intentionally not inlined into text (they would leak into retrieved context). - Chunk —
TextChunkerwraps LangChain'sRecursiveCharacterTextSplitter(with a dependency-free fallback), preferring paragraph/sentence boundaries before hard cuts. Chunk size/overlap come from settings (1000/150). - Embed —
EmbeddingProvider.embed_documents()in batches ofEMBEDDING_BATCH_SIZE; counts are validated against chunk counts. - Persist — vectors are written to Chroma under deterministic ids
(
"{document_id}:{chunk_index}") with metadata (user_id,document_id,document_name,chunk_index,collection, counts…). Any previous version of the document is deleted first (idempotent re-index). A Postgres chunk ledger (document_chunks) is updated by the task for auditability. - Report — progress (
5 → 45 → 80 → 95 → 100) is written to the task row as it runs, and the document transitionspending → processing → ready|failed.
Retrieval (app/rag/retriever.py):
- Embed the query with the same provider.
- Ask Chroma for
max(top_k, RETRIEVAL_FETCH_K)neighbours under a where clause enforcing tenant isolation:{"user_id": {"$eq": <uid>}}ANDoptionaldocument_id ∈ scopeANDoptionalcollection. - Apply
RETRIEVAL_SCORE_THRESHOLD, return the toptop_kasRetrievedChunk. A wider candidate pool (fetch_k) stops one document's high-scoring chunks from crowding out the rest. - If nothing is returned while the store is non-empty, a loud warning is logged (the classic "vectors written by another process are invisible" failure mode).
Generation (app/rag/chain.py): RAGChain composes retriever + prompt +
LLM.
PromptBuilderbuilds a chat prompt: a system prompt (answer only from context, plain prose, no lists/quotes/titles), optional trimmed history (lastmax_history_turns), then the retrieved context as numbered, citable blocks plus the question. Context is capped atmax_context_chars(~32k chars).- Answer post-processing (
clean_answer/clean_answer_stream) is a deterministic cleanup pass that runs after generation, because small models ignore formatting instructions: it strips list markers, inline enumerations, bold/emphasis marks, leading "Based on the provided information…" clauses, trailing "Sources used:" lines and bare URLs, and removes near-duplicate sentences. The streaming variant cleans each finished sentence as it arrives so the first words reach the client fast. - Citations are parsed from the answer's bracketed
[n]context references and mapped back to 0-based source offsets.
Conversation persistence — ChatService writes both the user turn and the
assistant turn (with sources stored as a JSON envelope) to PostgreSQL, so history
and citations survive across requests. A conversation can persist a document
scope (settings_json); a stale scope (all referenced documents deleted) falls
back to searching the whole knowledge base rather than returning nothing.
app/llm/base.py defines the LLMProvider ABC — the RAG pipeline depends only on
this interface, never a concrete implementation. It exposes:
generate(messages, max_new_tokens?, temperature?, top_p?) → GenerationResultgenerate_stream(...) → Iterator[str](default: yield the whole completion once)health(),warmup(),count_tokens(text)
TransformerLLMProvider (transformers_llm.py) loads
Qwen/Qwen2.5-0.5B-Instruct lazily and caches (tokenizer, model, device, quantized) at module scope so every provider instance shares one loaded model.
- Device — from
LLM_DEVICE(cpu/cuda/mps/auto).autoprobes CUDA, then MPS, else CPU. - Threads — CPU-only deployments set
torch.set_num_threadsto all logical cores (orLLM_NUM_THREADS) since each token is a sequence of matmuls. - Precision —
LLM_TORCH_DTYPE(defaultfloat32, full precision, broadest CPU kernel coverage).low_cpu_mem_usagestreams the checkpoint in. - Attention —
LLM_ATTN_IMPLEMENTATION(defaultsdpa); the loader clears the model's sliding window before construction sosdpais safe and warning-free. - Generation controls —
LLM_TEMPERATURE,LLM_TOP_P,LLM_REPETITION_PENALTY,LLM_NO_REPEAT_NGRAM_SIZE(hard block that stops the 0.5B model looping a phrase until it runs out of tokens). - Budget trim — the provider clamps
max_new_tokenssoprompt_tokens + completion ≤ LLM_CONTEXT_WINDOW_TOKENS. - Streaming —
generate_streamruns the blockingmodel.generateon a worker thread and yields from a Hugging FaceTextIteratorStreamer(skip_prompt=True), so decoded pieces arrive as produced. warmup()is called from the app lifespan and the worker bootstrap to pay the cold-start cost up front.
StubLLMProvider (stub.py) is a deterministic, offline, dependency-free
generator used when LLM_PROVIDER=stub (tests/CI). It does not "think" — it
quotes the numbered context blocks back (one line per source) so the whole
pipeline is exercised without downloading a model. It is never for real
answers.
get_llm_provider() / reset_llm_provider() manage the process-wide singleton.
PostgreSQL schema (SQLAlchemy 2.0 declarative). Every table uses a string UUID
primary key; users, documents and conversations also carry
created_at/updated_at via TimestampMixin. Foreign keys cascade on delete
(ondelete="CASCADE").
users—id,email(unique, indexed),full_name,hashed_password,role(user/admin),is_active,last_login_at.roleis a plain string (adding roles needs no migration). Indexix_users_email_lower.documents—owner_id→users,filename,original_filename,content_type,extension,size_bytes,checksum,status(pending/processing/ready/failed),error_message,storage_path,chunk_count,token_count,indexed_at,collection,tags. Indexix_documents_owner_status.document_chunks— the durable chunk ledger mirroring Chroma:document_id→documents,chunk_index,vector_id,content,char_count,token_count,created_at. Indexix_document_chunks_document_index.conversations—owner_id→users,title,is_archived(0/1 integer),message_count,settings_json(per-thread RAG overrides incl. document scope),last_message_at. Indexix_conversations_owner_updated.messages—conversation_id→conversations,role,content,sources(JSON envelope of citations),token_count,latency_ms,model_name,created_at. Indexix_messages_conversation_created.background_tasks— durable Celery audit trail:owner_id,celery_task_id,name,queue,status(queued/running/succeeded/failed/revoked),resource_type,resource_id,progress,result_json,error_message,started_at,finished_at. Indexes onowner_id,celery_task_id,resource_id,status.
Schema management: app/db/init_db.py provides create_all() (dev/Docker/test
boot) and init_database() (also seeds a bootstrap admin in development when
DEFAULT_ADMIN_EMAIL/DEFAULT_ADMIN_PASSWORD are set). Alembic is declared for
production migrations.
Password hashing (core/security.py) — bcrypt directly (no passlib), 12
rounds. Passwords longer than bcrypt's 72-byte limit are SHA-256 pre-hashed first.
hash_password() enforces a minimum length of 8; verify_password() is
constant-time.
JWT — PyJWT, HS256, signed with SECRET_KEY. Access and refresh tokens
carry sub, type, iss, iat, nbf, exp, jti and scopes. Decoding
verifies signature, issuer and the required claims, and rejects a token whose
type does not match what the caller expects. Lifetimes: 30 minutes (access) /
7 days (refresh). AuthService.refresh() rotates the pair and re-checks that the
subject still exists and is active. Logout is stateless (tokens are not
revocable without a denylist); the client discards them.
Authorisation — get_current_user resolves the bearer token to a User,
rejecting missing/malformed tokens, unknown subjects, and inactive accounts.
get_optional_user allows anonymous callers (used by rate limiting).
require_admin guards admin-only routes.
Tenant isolation — every retrieval is scoped to the caller's user_id via a
Chroma where clause, and every document/conversation/task query is filtered by
owner_id. Users can only ever see their own data.
Upload policy — extension allow-list, streamed size ceiling
(MAX_UPLOAD_SIZE_MB), non-empty content, and normalised extensions.
Transport/response hardening — SecurityHeadersMiddleware sets
X-Content-Type-Options: nosniff, X-Frame-Options: DENY,
Referrer-Policy: no-referrer. CORS is restricted to CORS_ORIGINS with
credentials enabled.
Error privacy — the login path returns one identical error for unknown-email
and wrong-password (no account enumeration). Database errors are logged
server-side but surface only as a generic storage_error.
A from-scratch sliding-window-log limiter (core/rate_limit.py) — no
slowapi/limits dependency. Each request is recorded in a Redis sorted set
keyed ratelimit:{scope}:{identity} (score = timestamp in µs); counting members
inside the window yields an exact sliding window. The read/prune/count/write
sequence runs in a single atomic MULTI/EXEC pipeline so concurrent workers
cannot over-admit. When Redis is unconfigured or unreachable, a thread-safe
in-memory store is used instead (keeps tests infrastructure-free and makes a
Redis blip degrade rate limiting rather than fail requests).
- Scopes —
default(60/min),auth(10/min),chat(20/min),upload(10/min), all configurable; window isRATE_LIMIT_WINDOW_SECONDS. - Identity —
user:<id>when authenticated, elseip:<client-ip>(X-Forwarded-For→X-Real-IP→ socket peer). - Enforcement — dependencies in
api/rate_limit_dep.pyconsume a unit and stash theRateLimitResultonrequest.state; the middleware copies it to the response asX-RateLimit-*. On breach, aRateLimitError→429withRetry-AfterandX-RateLimit-*headers. - Health endpoints are rate-limited too, guaranteeing
X-RateLimit-*on every response the API returns.
workers/celery_app.py configures Celery with Redis as broker/backend, JSON
serialisation, task_track_started, acks_late, reject_on_worker_lost,
prefetch=1, max_tasks_per_child=50, UTC timezone and a 20m soft / 25m hard
time limit. Queues: default, ingestion (ingestion tasks) and maintenance.
celery_app.set_default() registers it as the process-wide default so
shared_task resolves correctly from any thread (TestClient, threaded servers).
Tasks (workers/tasks/):
ingestion.ingest_document(document_id, task_record_id?)— the full RAG ingestion run; marks the document and task recordprocessing/running, writes progress, and finishes withready/succeededorfailed. Retries up to 3× with a 15s delay. Owns its own DB session(s) viasession_scope().maintenance.reindex_document(document_id, task_record_id?)— resets a document topendingand re-runs ingestion.maintenance.cleanup_stale_tasks(max_age_hours=24)— fails task rows stuck inqueued/runningpast the cutoff.maintenance.health_check()— trivial round-trip probe reporting LLM and embedding health.
Durability — Celery results live in Redis (a cache), so every dispatch is
also recorded as a BackgroundTask row. TaskService.get_task() refreshes the
durable row from Celery state while a task is in-flight and never commits stale
state.
Run a worker from the same image:
celery -A app.workers.celery_app:celery_app worker --loglevel=infoEvery failure (application error, validation error, HTTP error, DB error, or an
unexpected exception) is serialised by api/errors.py to the identical shape
(ErrorResponse):
{
"code": "email_taken",
"message": "An account with this email already exists.",
"details": null,
"request_id": "3f2a9c…"
}| Handler | Status | code |
|---|---|---|
AppError (base of the domain hierarchy) |
per-exception | domain code (e.g. email_taken, conversation_not_found, file_too_large) |
RequestValidationError |
422 |
validation_error (details = field errors) |
StarletteHTTPException |
as raised | http_<status> |
SQLAlchemyError |
503 |
storage_error |
Exception |
500 |
internal_error |
RateLimitError additionally sets Retry-After and X-RateLimit-*. Error
details are JSON-coerced with default=str so Pydantic validation contexts
(which embed exception objects) never break serialisation.
Domain exceptions (core/exceptions.py): AppError → ValidationError,
NotFoundError, ConflictError, AuthenticationError, PermissionDeniedError,
RateLimitError, StorageError → VectorStoreError, EmbeddingError,
LLMError, DocumentProcessingError, FileValidationError, TaskError.
Next.js 16 App Router, React 19, Tailwind CSS v4. The console is a
client-rendered SPA shell: "use client" components hold tokens in
localStorage and drive the UI; the API remains the authority.
src/app/ (App Router):
| Route | File | Access | Purpose |
|---|---|---|---|
/ |
app/page.js |
public | Marketing landing page; redirects authenticated users to /app/chat |
/login |
app/login/page.js |
public | Sign-in form (email + password) |
/register |
app/register/page.js |
public | Account creation (name, email, password with strength checks) |
/app |
app/app/page.js |
protected | Redirects to /app/chat |
/app/chat |
app/app/chat/page.js |
protected | Chat console: SSE-streamed answers + citations |
/app/documents |
app/app/documents/page.js |
protected | Upload, monitor, inspect chunks, edit/reindex/delete |
/app/dashboard |
app/app/dashboard/page.js |
protected | Counters, dependency health, task audit trail |
/app/settings |
app/app/settings/page.js |
protected | Profile, appearance, account controls |
| (404) | app/not-found.js |
public | Global 404 |
| (error) | app/app/error.js |
protected | Route error boundary |
| (loading) | app/app/loading.js |
protected | Route-level loading UI |
Root layout (app/layout.js) sets metadata, loads Geist fonts, wraps the tree
in ThemeProvider → AuthProvider → ToastProvider, and injects a tiny pre-paint
script that applies the stored theme to <html> (avoiding a light→dark flash).
Console layout (app/app/layout.js) is the protected shell: it renders the
Sidebar + TopBar and an AuthGate that redirects unauthenticated users to
/login. Crucially, {children} is always rendered and the gate draws a
blocking overlay on top instead of returning early — a layout that returns
before rendering {children} would drop every segment in its subtree (Next.js
"instant navigation" validation error).
Path alias — @/* → ./src/* (see jsconfig.json).
src/lib/auth.js exposes AuthProvider + useAuth():
- Bootstrap — on mount, read the cached user and token from
localStorage. If a token exists, optimistically set the cached user, then callGET /auth/meto confirm (this also exercises token refresh). On failure, clear storage. - login(email, password) —
POST /auth/login, store theTokenPair, set user. - register({email, password, full_name}) —
POST /auth/registerthenPOST /auth/login(auto sign-in), store tokens. - logout() — call
POST /auth/logout(best-effort), clear storage, drop user. - refreshUser() — re-fetch
GET /auth/meand update the cached user.
The context value is { user, loading, isAuthenticated, login, register, logout, refreshUser }.
src/lib/api.js is the single backend client.
- Base URL —
NEXT_PUBLIC_API_BASE_URL(defaulthttp://localhost:8000) + fixed prefix/api/v1. - Token storage (
tokenStore) — keysrag.access_token,rag.refresh_token,rag.userinlocalStorage(SSR-safe guards). request(path, { method, body, headers, auth, retry })— JSON-encodes non-FormDatabodies, attachesAuthorization: Bearer <access>whenauth, and on a401performs one transparent refresh then retries once. On failure it throws anApiErrorcarrying{ status, code, details }parsed from the backend's error envelope.204→null; non-JSON → text.refreshAccessToken()— de-duplicates concurrent refreshes via a shared promise; on failure clears the token store.api— thin typed wrappers grouped by domain:auth,users(profile, stats, deactivate),documents(list/stats/get/status/chunks/upload/update/ reindex/delete),chat(ask, conversations CRUD),tasks,health.
src/lib/sse.js consumes POST /chat/stream manually via fetch +
ReadableStream (the browser EventSource cannot send an Authorization header
nor POST).
parseEventStream(stream)— an async generator that decodes UTF-8, splits on the blank-line frame boundary, and parsesevent:/data:lines (:comment lines ignored). Each yielded frame is{ event, data }with JSON-parsed data.streamChat(payload, handlers, signal)— POSTs the payload with the bearer token andAccept: text/event-stream, then dispatches frames toonStart/onSources/onToken/onDone/onError. Supports anAbortSignalso the UI can stop generation. Non-OK responses are surfaced as anonErrorwith the backend message.
ThemeProvider(lib/theme.js) —light/dark/system, persisted underrag.theme.applyThemetoggles thedarkclass on<html>, which the@custom-variant dark (&:where(.dark, .dark *))rule inglobals.csskeys off (Tailwind v4 defaults toprefers-color-scheme, which a user control can't override). Follows OS changes while insystemmode.ToastProvider(lib/toast.js) — dependency-free{ success, error, info, dismiss }notifications rendered as a fixed stack (auto-dismiss timers; errors live longer).- Styling (
app/globals.css) —@import "tailwindcss", a--brand-*indigo scale exposed via@theme inline, thin scrollbars, and fade-in/slide-in/shimmer keyframes (respectingprefers-reduced-motion).
components/ui.js— presentational primitives:Card,CardHeader,BrandMark,PageHeader,StatCard,Badge,Button,Input,Select,Checkbox,SearchInput,Alert,EmptyState,Spinner,Skeleton,ProgressBar,Modal,CopyButton,StatusBadge,TaskStatusBadge, and formattersformatBytes/formatDate/formatRelative/formatNumber.components/shell.js— console chrome:NAV(Chat, Documents, Dashboard, Settings),Sidebar,TopBar,ThemeSwitcher,UserMenu.components/icons.js— inline SVG icon set (no icon dependency).components/auth-shell.js— the two-column auth layout (brand panel + form card) used by login/register.
Page highlights
- Chat — conversation rail + transcript; live tokens via
streamChat; source chips (cited ones highlighted) with a Stop button; scope selector and "include history" toggle. - Documents — drag-and-drop
UploadZone, status filters, search, pagination, aChunkViewermodal, and edit/reindex/delete actions with confirmation. - Dashboard — stat tiles, a "Service health" panel (per-dependency readiness), a "Background tasks" list with progress bars, and a worker health probe button.
- Settings — profile form (
PATCH /users/me), theme switcher, account lifecycle (sign out, deactivate), and an "about this deployment" card.
- Python 3.11
- Node.js 22 (frontend)
- PostgreSQL (required — there is no SQLite fallback in the app)
- Redis (cache, rate limits, Celery broker)
- ~2 GB disk for the model + embedding weights (downloaded on first use)
cd backend
# (recommended) create a virtual environment
python -m venv .venv && source .venv/bin/activate
# CPU-only PyTorch first (the default index pulls CUDA wheels)
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install -r requirements.txt
# review/point at your services (PostgreSQL + Redis)
# DATABASE_URL, REDIS_URL, SECRET_KEY, CORS_ORIGINS
$EDITOR .env
# run the API
uvicorn app.main:app --reload --port 8000- API: http://localhost:8000
- Swagger UI: http://localhost:8000/docs
- ReDoc: http://localhost:8000/redoc
The schema is created automatically on startup (init_database). Tables are also
created under PostgreSQL first boot.
Offline / quick start. To run without downloading any model weights, set
LLM_PROVIDER=stubandEMBEDDING_PROVIDER=hashin.env. The stub answers extractively from retrieved context (fine for smoke tests, not real answers).
cd frontend
npm install
NEXT_PUBLIC_API_BASE_URL=http://localhost:8000 npm run devOpen http://localhost:3000, register an account, upload a document, and chat.
The frontend origin must be present in the backend's
CORS_ORIGINS(defaults already includehttp://localhost:3000).
With CELERY_TASK_ALWAYS_EAGER=true (the default) ingestion runs inside the API
process — no worker is required. To run a real worker instead, set
CELERY_TASK_ALWAYS_EAGER=false and start it from the same environment:
cd backend
celery -A app.workers.celery_app:celery_app worker --loglevel=infoThe suite is hermetic — no PostgreSQL, no Redis, no model downloads required:
cd backend
pytest # full run
pytest tests/unit # unit only
pytest tests/integration # integration (API) only
pytest --cov=app # with coveragetests/conftest.py configures the environment before importing the app and
swaps in: SQLite (temp file, NullPool), in-memory cache, eager Celery,
stub LLM + hash embeddings, and an embedded Chroma store in a temp
directory. Test layout:
tests/unit/— security, rate limiting, chunking/loading, prompt/chain, vector store/retriever, ingestion pipeline, LLM/embeddings.tests/integration/— auth, documents, chat (incl. SSE), tasks, health + cross-cutting middleware/OpenAPI.
- Layered:
api → services → repositories/models, withrag/,llm/,db/,workers/as infrastructure. Services own transactions; repositories never commit. - Singletons (
get_*/reset_*) are used for heavy process-wide resources and are reset between tests. - All external calls are behind a provider interface with a deterministic test double.
cd frontend
npm run dev # dev server (Turbopack)
npm run build # production build (standalone output)
npm start # serve the production buildConfig: next.config.mjs enables output: "standalone", React strict mode,
cacheComponents, partialPrefetching, and the Tailwind v4 Turbopack loader.
Both services ship with multi-stage Dockerfiles; the backend image runs both
the API and the Celery worker so they share one pinned dependency set. There is
no docker-compose.yml in the repository, so you provide the orchestration
(compose, Kubernetes, etc.) and the PostgreSQL + Redis services.
- Builder —
python:3.11-slim, installs the build toolchain, a venv, CPU-only PyTorch andrequirements.txt, then optionally pre-downloads the embedding and LLM weights (PREFETCH_MODELS=trueby default) into/opt/hf. - Runtime — slim image with only the venv + HF cache + app, runs as a
non-root user (
appuser),tinias PID 1. Data dirs:/data/chroma,/data/uploads,/data/hf.HEALTHCHECKhits/api/v1/health. - Default CMD — Gunicorn +
UvicornWorker, 2 workers,--timeout 900(covers worst-case CPU generation and the SSE keep-alive). - Worker — override the command:
celery -A app.workers.celery_app:celery_app worker --loglevel=info.
Build args/environment: PREFETCH_MODELS, LLM_MODEL_ID, EMBEDDING_MODEL.
- Multi-stage
node:22-alpine: deps (npm ci) → build (npm run build) → runner serving the standalone output (node server.js) as a non-root user. NEXT_PUBLIC_API_BASE_URLis baked at build time (browser-visible) — pass the public backend URL as a build arg. Listens on0.0.0.0:3000;HEALTHCHECKfetches/.
- Set a strong
SECRET_KEY(python -c "import secrets; print(secrets.token_urlsafe(64))"). - Set
APP_ENV=production. - Restrict
CORS_ORIGINSto your real frontend origin(s). - Use managed PostgreSQL and Redis; tune
DB_POOL_*andREDIS_URL. - Serve over TLS (terminate at a reverse proxy; keep
X-Accel-Buffering: noso SSE is not buffered). - Allocate enough CPU/RAM for local inference (the 0.5B model loads in fp32 by
default; use
LLM_TORCH_DTYPE=float16to halve memory). - Decide on worker topology: single process (
CELERY_TASK_ALWAYS_EAGER=true) vs. separate worker + shared Chroma persistence. - Apply Alembic migrations (or
create_all()) against the production DB. - Persist
/data(Chroma + uploads + HF cache) on durable volumes.
No license file is included in this repository. Add one (e.g. MIT/Apache-2.0) or set an explicit license before distributing.