Skip to content
spark-369Public

About

Upload your own documents (PDF, DOCX, Markdown, CSV, JSON, HTML, TXT), and ask questions in natural language. Answers are generated by a local language model, grounded in the retrieved chunks from your documents, and returned with citations back to the exact source passages — everything runs on your own infrastructure, with no third-party API calls

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

2 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 

Repository files navigation

Enterprise RAG Knowledge Assistant

A self-hosted, domain-specific Retrieval-Augmented Generation (RAG) knowledge assistant. Upload your own documents (PDF, DOCX, Markdown, CSV, JSON, HTML, TXT), and ask questions in natural language. Answers are generated by a local language model, grounded in the retrieved chunks from your documents, and returned with citations back to the exact source passages — everything runs on your own infrastructure, with no third-party API calls.

Backend FastAPI (Python 3.11) · SQLAlchemy 2.0 · PostgreSQL · Redis · Celery · ChromaDB · Hugging Face Transformers (Qwen2.5-0.5B-Instruct) · LangChain
Frontend Next.js 16 (App Router) · React 19 · Tailwind CSS v4
Delivery JSON REST API + Server-Sent Events (SSE) token streaming

Table of contents

  1. Key features
  2. High-level architecture
  3. Repository layout
  4. Technology stack
  5. Backend documentation
  6. Frontend documentation
  7. Getting started
  8. Development & testing
  9. Deployment
  10. License

Key features

  • Async ingestion pipeline — uploads are validated, stored and dispatched to a Celery worker (or run inline) that parses, chunks, embeds and indexes the document, tracking progress in a durable task ledger.
  • Embedded, persistent vector search — ChromaDB runs inside the Python process and persists to disk; no separate search service to operate.
  • Per-user isolation — every vector carries its owner's user_id; retrieval injects a Chroma where clause so a user can never see another tenant's chunks.
  • Local LLM inference — grounded answers from Qwen2.5-0.5B-Instruct served through transformers, plus a deterministic stub provider for tests/CI.
  • Real token streaming — answers arrive over SSE as the model produces them (start → sources → token* → done), rendered live in the console.
  • Citations — each answer carries the retrieved source chunks (with snippet and similarity score) that grounded it.
  • JWT auth with refresh — access/refresh token pair, transparent client-side refresh, scope-aware verification.
  • Manual, scope-aware rate limiting — a hand-written sliding-window-log limiter (Redis-backed, in-memory fallback) protecting auth, chat, upload and general traffic.
  • Uniform error envelope — every failure returns the same JSON shape (code, message, details, request_id).
  • Observability — request-id correlation, structured access logs, X-Request-ID, X-Process-Time-Ms and X-RateLimit-* response headers, plus liveness/readiness probes that check every dependency.
  • Dark/light/system theming and a responsive, dependency-light UI.

High-level architecture

┌──────────────────────────────────────────────────────────────────────────┐
│                              Browser                                      │
│   Next.js 16 App Router (React 19)  ·  Tailwind CSS v4                    │
│   ┌──────────────┐  ┌───────────────┐  ┌───────────────┐  ┌────────────┐  │
│   │  Chat        │  │  Documents    │  │  Dashboard    │  │  Settings  │  │
│   └──────────────┘  └───────────────┘  └───────────────┘  └────────────┘  │
└───────────────────────────────┬──────────────────────────────────────────┘
        fetch + Bearer JWT       │   SSE (fetch + ReadableStream)
                                 ▼
┌──────────────────────────────────────────────────────────────────────────┐
│                        FastAPI application (backend/)                     │
│  Middleware: RequestContext → RateLimitHeaders → Security → GZip → CORS   │
│  Routers:  /api/v1/{health, auth, users, documents, chat, tasks}          │
│                                                                           │
│  Services  ── DocumentService · ChatService · AuthService · TaskService   │
│  RAG       ── RAGChain · Retriever · PromptBuilder · Ingestion Pipeline   │
│  LLM       ── TransformersLLMProvider (Qwen2.5-0.5B) · StubLLMProvider    │
│  Stores    ── SQLAlchemy (PostgreSQL) · ChromaDB (embedded) · Redis cache │
└───────┬─────────────────────┬───────────────────────┬─────────────────────┘
        │                     │                        │
        ▼                     ▼                        ▼
  PostgreSQL             Redis                 ChromaDB (on-disk)
  users, documents,      cache, rate-limit     knowledge_base collection
  chunks, conversations, counters, Celery       (vectors + metadata)
  messages, tasks        broker + backend

        ▲
        │ consume "ingestion" / "maintenance" queues
┌───────┴──────────────────────────────────────────────────────────────────┐
│                      Celery worker (same image)                           │
│  ingest_document  ·  reindex_document  ·  cleanup_stale_tasks  ·  health   │
└──────────────────────────────────────────────────────────────────────────┘

Design principles

  • Layered backend — HTTP (api/) → business logic (services/) → persistence (repositories/ + models/) → infrastructure (db/, workers/, rag/, llm/). Services own transactions; repositories never commit.
  • Provider abstraction — both the LLM (LLMProvider) and embeddings (EmbeddingProvider) are interfaces with a production implementation and a deterministic, dependency-free one for tests, selected by configuration.
  • Lazy singletons — the vector store, retriever, embedding provider, LLM provider, cache, rate limiter and ingestion pipeline are built once per process behind a lock, so heavy models load at most once.
  • Client is a client-rendered SPA shell — tokens live in localStorage; the API is the single source of truth for authorisation.

Repository layout

v1/
├── README.md                    # This file
├── backend/                     # FastAPI service + Celery worker
│   ├── Dockerfile               # Multi-stage image (API + worker share it)
│   ├── requirements.txt         # Python dependencies (range-pinned)
│   ├── .env                     # Committed backend configuration (source of truth)
│   ├── app/
│   │   ├── main.py              # Application factory + lifespan + middleware
│   │   ├── api/                 # HTTP layer (routers, deps, middleware, errors)
│   │   ├── core/                # Config, security, logging, rate limiting, exceptions
│   │   ├── db/                  # Engine/session, cache, vector store, init
│   │   ├── llm/                 # LLMProvider interface, transformers + stub impls, prompts
│   │   ├── models/              # SQLAlchemy ORM models
│   │   ├── rag/                 # Loaders, chunking, embeddings, retriever, chain, pipeline
│   │   ├── repositories/        # Data-access layer (CRUD, no commits)
│   │   ├── schemas/             # Pydantic request/response models
│   │   ├── services/            # Business logic (auth, documents, chat, tasks)
│   │   └── workers/             # Celery app + ingestion & maintenance tasks
│   ├── tests/                   # Pytest: unit/ and integration/
│   └── data/                    # Runtime artefacts (chroma, uploads, hf cache)
└── frontend/                    # Next.js console
    ├── Dockerfile               # Multi-stage (standalone) image
    ├── package.json             # next 16 · react 19 · tailwindcss v4
    ├── next.config.mjs          # output: "standalone", turbopack, cacheComponents
    ├── jsconfig.json            # "@/*" → "./src/*" path alias
    └── src/
        ├── app/                 # App Router routes (public + /app console)
        ├── components/           # ui.js, shell.js, icons.js, auth-shell.js
        └── lib/                 # api.js, sse.js, auth.js, theme.js, toast.js

Technology stack

Backend

Concern Choice Notes
Web framework FastAPI >=0.115,<0.116 Async, OpenAPI, dependency injection
ASGI server Gunicorn + Uvicorn workers UvicornWorker, 2 workers, 900s timeout
Validation Pydantic v2 + pydantic-settings Typed settings + request/response schemas
ORM / DB SQLAlchemy 2.0 + psycopg2 PostgreSQL only (no SQLite in production)
Migrations Alembic (declared) create_all() used for local/dev boot
Auth PyJWT + bcrypt HS256 JWTs, bcrypt password hashing (12 rounds)
Cache/queue Redis 6.1+ + Celery 5.4 Cache, rate limits, broker + result backend
Vector store ChromaDB >=0.5.20,<0.7 Embedded + persistent, no server
RAG tooling LangChain 0.3.x Recursive splitter, Document interop
Inference transformers + torch (CPU) Qwen2.5-0.5B-Instruct, full-precision fp32
Embeddings sentence-transformers all-MiniLM-L6-v2 (384-dim)
Parsing pypdf, python-docx, beautifulsoup4 PDF/DOCX/HTML (+ CSV/JSON/text built in)
Testing pytest, pytest-asyncio, pytest-cov, httpx Hermetic (no external services)

Frontend

Concern Choice Notes
Framework Next.js 16 (App Router) output: "standalone" for Docker
UI runtime React 19 Client components ("use client")
Styling Tailwind CSS v4 @import "tailwindcss", @custom-variant dark
Fonts next/font (Geist, Geist Mono) Self-hosted via next/font/google
State React Context (auth, theme, toast) No external state library
Transport fetch + manual SSE parser EventSource can't POST/send headers

Backend documentation

Package structure

app/
├── main.py                  # create_app(), lifespan (init DB, warm LLM), middleware, OpenAPI
├── api/
│   ├── dependencies.py      # get_current_user, get_optional_user, require_admin, pagination
│   ├── rate_limit_dep.py    # rate_limit_default / _auth / _chat / _upload
│   ├── middleware.py        # RequestContext, RateLimitHeaders, SecurityHeaders
│   ├── errors.py            # Uniform exception handlers
│   └── v1/
│       ├── router.py        # Mounts every endpoint module under /api/v1
│       └── endpoints/       # health.py, auth.py, users.py, documents.py, chat.py, tasks.py
├── core/
│   ├── config.py            # Settings (pydantic-settings) + LLM budget constants
│   ├── security.py          # bcrypt hashing, JWT issue/verify
│   ├── rate_limit.py        # Sliding-window limiter (Redis + in-memory stores)
│   ├── logging.py           # configure_logging, RequestContextFilter, get_logger
│   └── exceptions.py        # AppError hierarchy (HTTP status + stable code)
├── db/
│   ├── base.py              # Declarative Base + UUID/Timestamp mixins
│   ├── session.py           # Engine/session factory, get_db, session_scope, configure_engine
│   ├── cache.py             # CacheClient (Redis) + InMemoryCache fallback
│   ├── vector_store.py      # VectorStore over Chroma (telemetry disabled)
│   └── init_db.py           # create_all(), seed_default_admin(), init_database()
├── llm/
│   ├── base.py              # LLMProvider ABC, Message, GenerationResult
│   ├── factory.py           # build/get/reset_llm_provider (singleton)
│   ├── transformers_llm.py  # Qwen2.5-0.5B inference (+ streaming)
│   ├── stub.py              # Deterministic extractive provider (tests)
│   └── prompt.py            # PromptBuilder (system + context + history)
├── models/                  # user.py, document.py, conversation.py, task.py
├── rag/
│   ├── loaders.py           # DocumentLoader (bytes → text + metadata)
│   ├── chunking.py          # TextChunker (LangChain recursive splitter)
│   ├── embeddings.py        # EmbeddingProvider: huggingface + hash
│   ├── retriever.py         # Retriever (query embed + Chroma search + threshold)
│   ├── chain.py             # RAGChain + answer post-processing (clean_answer)
│   └── pipeline.py          # IngestionPipeline (load → chunk → embed → persist)
├── repositories/            # base.py, user.py, document.py, conversation.py, task.py
├── schemas/                 # common.py, auth.py, document.py, chat.py, task.py
├── services/                # auth_service.py, document_service.py, chat_service.py, task_service.py
└── workers/
    ├── celery_app.py        # Celery app (queues, eager flag, set_default)
    └── tasks/               # ingestion.py, maintenance.py

Request lifecycle

  1. Middleware chain (outermost first): RequestContextMiddleware assigns/propagates X-Request-ID, times the request and logs the access line → RateLimitHeadersMiddleware copies any limiter result onto the response → SecurityHeadersMiddleware adds hardening headers → GZipMiddleware → CORSMiddleware.
  2. Routing — the request hits a router mounted under /api/v1.
  3. Dependencies — auth (get_current_user), DB session (get_db), scope rate-limit (rate_limit_*) and pagination (pagination_params) are resolved.
  4. Service layer — the endpoint calls a service, which drives repositories and/or the RAG chain and owns the transaction (session.commit()).
  5. Response — a Pydantic schema is serialised; errors are converted by the central handlers into the uniform error envelope.

Startup (lifespan) — configure logging, ensure the schema exists (init_database, non-fatal if unavailable), create the upload/Chroma directories, and warm the LLM weights (transformers only, skipped in tests, never fatal). Shutdown disposes the engine only if this process created it.

Configuration reference

All settings live in app/core/config.py (Settings) and are read from backend/.env (committed) with an optional repo-root .env override. List values (CORS_ORIGINS, ALLOWED_EXTENSIONS) accept JSON arrays or comma-separated strings.

Variable Default Purpose
APP_NAME Enterprise RAG Knowledge Assistant Display name
APP_ENV development development/staging/production/test
DEBUG false FastAPI debug mode
LOG_LEVEL INFO Root log level
API_V1_PREFIX /api/v1 API mount prefix
CORS_ORIGINS ["http://localhost:3000","http://127.0.0.1:3000"] Allowed browser origins
SECRET_KEY (placeholder) JWT signing key — change in production
JWT_ALGORITHM HS256 JWT algorithm
JWT_ISSUER rag-knowledge-assistant iss claim, verified on decode
ACCESS_TOKEN_EXPIRE_MINUTES 30 Access-token lifetime
REFRESH_TOKEN_EXPIRE_DAYS 7 Refresh-token lifetime
RATE_LIMIT_ENABLED true Master switch for the limiter
RATE_LIMIT_REQUESTS / _WINDOW_SECONDS 60 / 60 Default scope
RATE_LIMIT_AUTH_REQUESTS 10 Auth scope (brute-force protection)
RATE_LIMIT_CHAT_REQUESTS 20 Chat scope (expensive path)
RATE_LIMIT_UPLOAD_REQUESTS 10 Upload scope
DATABASE_URL postgresql+psycopg2://rag:rag@localhost:5432/rag PostgreSQL DSN
DB_ECHO / DB_POOL_SIZE / DB_MAX_OVERFLOW / DB_POOL_PRE_PING false/5/10/true Engine tuning
REDIS_URL redis://localhost:6379/0 Cache/broker/limits
CHROMA_PERSIST_DIR ./data/chroma On-disk vector store
CHROMA_COLLECTION knowledge_base Default collection name
UPLOAD_DIR ./data/uploads Stored uploads
MAX_UPLOAD_SIZE_MB 25 Upload ceiling
ALLOWED_EXTENSIONS .pdf,.txt,.md,.csv,.docx,.json,.html Accepted file types
EMBEDDING_PROVIDER huggingface huggingface or hash
EMBEDDING_MODEL sentence-transformers/all-MiniLM-L6-v2 Embedding model
EMBEDDING_DIMENSION 384 Vector dimension
EMBEDDING_BATCH_SIZE / EMBEDDING_NORMALIZE 32 / true Embedding tuning
CHUNK_SIZE / CHUNK_OVERLAP 1000 / 150 Splitter settings
RETRIEVAL_TOP_K / RETRIEVAL_FETCH_K 4 / 20 Context size / candidate pool
RETRIEVAL_SCORE_THRESHOLD 0.0 Min similarity (raise for semantic embeddings)
LLM_PROVIDER transformers transformers or stub
LLM_MODEL_ID Qwen/Qwen2.5-0.5B-Instruct Model to load
LLM_MAX_NEW_TOKENS 512 (env) / 32768 (ceiling) Generation budget
LLM_TEMPERATURE / LLM_TOP_P 0.0 / 0.9 Sampling
LLM_REPETITION_PENALTY / LLM_NO_REPEAT_NGRAM_SIZE 1.1 / 4 Anti-loop controls
LLM_DEVICE / LLM_NUM_THREADS cpu / 0 (all cores) Device & CPU threads
LLM_ATTN_IMPLEMENTATION sdpa sdpa / eager / flash_attention_2
LLM_TORCH_DTYPE float32 float32/float16/bfloat16/auto
LLM_LOW_CPU_MEM_USAGE / LLM_TRUST_REMOTE_CODE true / false Load options
CHAT_USE_CELERY false Route generation through Celery
HF_HOME ./data/hf Hugging Face cache dir
CELERY_BROKER_URL / CELERY_RESULT_BACKEND (empty → REDIS_URL) Celery endpoints
CELERY_TASK_ALWAYS_EAGER true Run tasks inline (see note below)
CELERY_WORKER_CONCURRENCY 2 Worker concurrency

Note on eager tasks. The embedded Chroma client keeps its ANN index in memory, so a separate worker process writing to the same persist directory would be invisible to the API process (retrieval would return nothing). The default CELERY_TASK_ALWAYS_EAGER=true runs ingestion inside the API process so the collection has a single owner. Set it to false only when the API and worker are separate containers and read-after-write visibility is acceptable to trade away.

Generation-budget constants (app/core/config.py):

  • LLM_CONTEXT_WINDOW_TOKENS = 32_768 — hard total budget (prompt + completion).
  • LLM_MAX_GENERATION_TOKENS = 32_768 — maximum tokens the model may emit; the provider trims it at request time to context_window − prompt_tokens.
  • llm_max_new_tokens is validated to stay within these ceilings at configuration time.

REST API reference

Base URL: /api/v1. All endpoints require a Bearer access token except /health*, /auth/register, /auth/login, and /auth/refresh. Interactive docs: /docs (Swagger UI) and /redoc; the raw schema is at /openapi.json (with a BearerAuth security scheme).

Common headers

Header Direction Meaning
Authorization: Bearer <jwt> request Access token
X-Request-ID both Correlation id (generated if absent, echoed back)
X-Process-Time-Ms response Server-side handling time
X-RateLimit-Limit / -Remaining / -Reset response Limiter state
Retry-After response (429) Seconds until the quota resets

Pagination — list endpoints accept page (≥1, ≤10000) and page_size (1–100, default 20) and return a Page<T> envelope: { items, total, page, page_size, total_pages, has_next }.

Health (tags: health)

Method Path Auth Rate scope Description
GET /health no default Cheap liveness probe
GET /health/ready no default Deep readiness: database, cache, broker, embeddings, LLM

Authentication (/auth)

Method Path Auth Rate scope Description
POST /auth/register no auth Create an account → 201 UserRead
POST /auth/login no auth Exchange credentials → TokenPair
POST /auth/refresh no auth Rotate a refresh token → TokenPair
GET /auth/me yes — Current user profile
POST /auth/logout yes — Stateless logout acknowledgement
POST /api/v1/auth/login
Content-Type: application/json

{ "email": "you@company.com", "password": "Password123" }
{
  "access_token": "eyJ...",
  "refresh_token": "eyJ...",
  "token_type": "bearer",
  "expires_in": 1800,
  "user": { "id": "…", "email": "you@company.com", "role": "user", "is_active": true }
}

Users (/users)

Method Path Auth Description
PATCH /users/me yes Update display name and/or password
GET /users/me/stats yes { documents: {...counts}, conversations, role }
DELETE /users/me yes Deactivate the account (soft delete)

Documents (/documents)

Method Path Auth Rate scope Description
POST /documents/upload yes upload Multipart upload → 202, dispatches ingestion
GET /documents yes — Paginated list; filters status, search, collection, extension
GET /documents/stats yes — Counts per status
GET /documents/{id} yes — Fetch one document
GET /documents/{id}/chunks yes — Paginated stored chunks
GET /documents/{id}/status yes — Ingestion progress (for polling)
PATCH /documents/{id} yes — Update metadata (name, tags, collection)
POST /documents/{id}/reindex yes upload Re-run ingestion → 202
DELETE /documents/{id} yes — Delete row + chunks + file + vectors

Upload form fields: file (required), collection (optional), tags (optional, comma-separated), process_now (bool, default true). The upload is streamed in ≤1 MiB chunks so the size ceiling is enforced while reading.

Chat (/chat)

Method Path Auth Rate scope Description
POST /chat/completions yes chat Non-streaming RAG answer
POST /chat/stream yes chat Stream the answer over SSE
POST /chat/stream/{conversation_id} yes chat Continue a thread and stream
POST /chat/conversations yes — Create a conversation → 201
GET /chat/conversations yes — List conversations (paginated)
GET /chat/conversations/{id} yes — Conversation + full message history
GET /chat/conversations/{id}/messages yes — Messages of one thread
PATCH /chat/conversations/{id} yes — Rename / archive
DELETE /chat/conversations/{id} yes — Delete thread + messages

ChatCompletionRequest fields: question (1–8000 chars), conversation_id?, document_ids? (scope retrieval), top_k? (1–20), temperature? (0–2), max_new_tokens? (16–32768), stream (ignored by SSE routes), use_history (default true).

Tasks (/tasks)

Method Path Auth Description
GET /tasks yes Paginated durable task audit trail
GET /tasks/{task_id} yes Fetch by record id or Celery id (refreshes from Celery)
POST /tasks/maintenance/health yes Dispatch a worker health probe → 202

Root

Method Path Description
GET / { name, version, docs, health } (outside the schema)

Server-Sent Events (SSE) contract

POST /api/v1/chat/stream (and /chat/stream/{conversation_id}) return text/event-stream. Each frame is a named event whose data is a JSON object with a type discriminator, matching ChatStreamEvent:

event: <type>
data: <json>

Frame sequence: start → sources → token* → done (or a terminal error).

Event Payload Meaning
start { conversation_id, model_name } Stream opened
sources { sources: SourceReference[] } Retrieved citations, sent before generation
token { text, index } One progressive answer fragment
done { message_id, conversation_id, answer, latency_ms, token_count, citations } Final answer (accumulated tokens) + metadata
error { message, code } Terminal failure (HTTP status is already 200)
heartbeat {} Keep-alive (ignored by the client)

SourceReference = { chunk_id, document_id, document_name, score, snippet, chunk_index, cited }.

Server notes. The stream owns its own database session. Because generation (and the SQLAlchemy session) is synchronous and CPU-bound, the service generator runs on a dedicated worker thread whose frames are marshalled to the event loop through an asyncio.Queue — this keeps the event loop unblocked and every DB call on a single thread. SSE headers disable proxy buffering (X-Accel-Buffering: no, Cache-Control: no-cache, no-transform).

event: start
data: {"conversation_id":"…","model_name":"Qwen/Qwen2.5-0.5B-Instruct"}

event: sources
data: {"sources":[{"chunk_id":"…:0","document_id":"…","document_name":"policy.pdf","score":0.83,"snippet":"…","chunk_index":0,"cited":false}]}

event: token
data: {"text":"Retention ","index":0}

event: token
data: {"text":"is 7 years…","index":1}

event: done
data: {"message_id":"…","conversation_id":"…","answer":"Retention is 7 years…","latency_ms":4120,"token_count":58,"citations":[0]}

RAG pipeline

Ingestion (app/rag/pipeline.py, run by the Celery worker or inline):

stored file ─▶ DocumentLoader ─▶ TextChunker ─▶ EmbeddingProvider ─▶ VectorStore (Chroma)
   (disk)        (bytes→text)     (recursive)      (batched embed)      (+ Postgres chunk ledger)
  1. Load — DocumentLoader dispatches on extension: PDF (pypdf), DOCX (python-docx, incl. tables), HTML (beautifulsoup4), CSV, JSON, or best-effort text decode. It normalises whitespace, extracts provenance metadata and rejects empty/no-text documents. Page numbers are intentionally not inlined into text (they would leak into retrieved context).
  2. Chunk — TextChunker wraps LangChain's RecursiveCharacterTextSplitter (with a dependency-free fallback), preferring paragraph/sentence boundaries before hard cuts. Chunk size/overlap come from settings (1000/150).
  3. Embed — EmbeddingProvider.embed_documents() in batches of EMBEDDING_BATCH_SIZE; counts are validated against chunk counts.
  4. Persist — vectors are written to Chroma under deterministic ids ("{document_id}:{chunk_index}") with metadata (user_id, document_id, document_name, chunk_index, collection, counts…). Any previous version of the document is deleted first (idempotent re-index). A Postgres chunk ledger (document_chunks) is updated by the task for auditability.
  5. Report — progress (5 → 45 → 80 → 95 → 100) is written to the task row as it runs, and the document transitions pending → processing → ready|failed.

Retrieval (app/rag/retriever.py):

  1. Embed the query with the same provider.
  2. Ask Chroma for max(top_k, RETRIEVAL_FETCH_K) neighbours under a where clause enforcing tenant isolation: {"user_id": {"$eq": <uid>}} AND optional document_id ∈ scope AND optional collection.
  3. Apply RETRIEVAL_SCORE_THRESHOLD, return the top top_k as RetrievedChunk. A wider candidate pool (fetch_k) stops one document's high-scoring chunks from crowding out the rest.
  4. If nothing is returned while the store is non-empty, a loud warning is logged (the classic "vectors written by another process are invisible" failure mode).

Generation (app/rag/chain.py): RAGChain composes retriever + prompt + LLM.

  • PromptBuilder builds a chat prompt: a system prompt (answer only from context, plain prose, no lists/quotes/titles), optional trimmed history (last max_history_turns), then the retrieved context as numbered, citable blocks plus the question. Context is capped at max_context_chars (~32k chars).
  • Answer post-processing (clean_answer / clean_answer_stream) is a deterministic cleanup pass that runs after generation, because small models ignore formatting instructions: it strips list markers, inline enumerations, bold/emphasis marks, leading "Based on the provided information…" clauses, trailing "Sources used:" lines and bare URLs, and removes near-duplicate sentences. The streaming variant cleans each finished sentence as it arrives so the first words reach the client fast.
  • Citations are parsed from the answer's bracketed [n] context references and mapped back to 0-based source offsets.

Conversation persistence — ChatService writes both the user turn and the assistant turn (with sources stored as a JSON envelope) to PostgreSQL, so history and citations survive across requests. A conversation can persist a document scope (settings_json); a stale scope (all referenced documents deleted) falls back to searching the whole knowledge base rather than returning nothing.

LLM inference layer

app/llm/base.py defines the LLMProvider ABC — the RAG pipeline depends only on this interface, never a concrete implementation. It exposes:

  • generate(messages, max_new_tokens?, temperature?, top_p?) → GenerationResult
  • generate_stream(...) → Iterator[str] (default: yield the whole completion once)
  • health(), warmup(), count_tokens(text)

TransformerLLMProvider (transformers_llm.py) loads Qwen/Qwen2.5-0.5B-Instruct lazily and caches (tokenizer, model, device, quantized) at module scope so every provider instance shares one loaded model.

  • Device — from LLM_DEVICE (cpu/cuda/mps/auto). auto probes CUDA, then MPS, else CPU.
  • Threads — CPU-only deployments set torch.set_num_threads to all logical cores (or LLM_NUM_THREADS) since each token is a sequence of matmuls.
  • Precision — LLM_TORCH_DTYPE (default float32, full precision, broadest CPU kernel coverage). low_cpu_mem_usage streams the checkpoint in.
  • Attention — LLM_ATTN_IMPLEMENTATION (default sdpa); the loader clears the model's sliding window before construction so sdpa is safe and warning-free.
  • Generation controls — LLM_TEMPERATURE, LLM_TOP_P, LLM_REPETITION_PENALTY, LLM_NO_REPEAT_NGRAM_SIZE (hard block that stops the 0.5B model looping a phrase until it runs out of tokens).
  • Budget trim — the provider clamps max_new_tokens so prompt_tokens + completion ≤ LLM_CONTEXT_WINDOW_TOKENS.
  • Streaming — generate_stream runs the blocking model.generate on a worker thread and yields from a Hugging Face TextIteratorStreamer (skip_prompt=True), so decoded pieces arrive as produced.
  • warmup() is called from the app lifespan and the worker bootstrap to pay the cold-start cost up front.

StubLLMProvider (stub.py) is a deterministic, offline, dependency-free generator used when LLM_PROVIDER=stub (tests/CI). It does not "think" — it quotes the numbered context blocks back (one line per source) so the whole pipeline is exercised without downloading a model. It is never for real answers.

get_llm_provider() / reset_llm_provider() manage the process-wide singleton.

Data model

PostgreSQL schema (SQLAlchemy 2.0 declarative). Every table uses a string UUID primary key; users, documents and conversations also carry created_at/updated_at via TimestampMixin. Foreign keys cascade on delete (ondelete="CASCADE").

  • users — id, email (unique, indexed), full_name, hashed_password, role (user/admin), is_active, last_login_at. role is a plain string (adding roles needs no migration). Index ix_users_email_lower.
  • documents — owner_id→users, filename, original_filename, content_type, extension, size_bytes, checksum, status (pending/processing/ready/failed), error_message, storage_path, chunk_count, token_count, indexed_at, collection, tags. Index ix_documents_owner_status.
  • document_chunks — the durable chunk ledger mirroring Chroma: document_id→documents, chunk_index, vector_id, content, char_count, token_count, created_at. Index ix_document_chunks_document_index.
  • conversations — owner_id→users, title, is_archived (0/1 integer), message_count, settings_json (per-thread RAG overrides incl. document scope), last_message_at. Index ix_conversations_owner_updated.
  • messages — conversation_id→conversations, role, content, sources (JSON envelope of citations), token_count, latency_ms, model_name, created_at. Index ix_messages_conversation_created.
  • background_tasks — durable Celery audit trail: owner_id, celery_task_id, name, queue, status (queued/running/succeeded/failed/revoked), resource_type, resource_id, progress, result_json, error_message, started_at, finished_at. Indexes on owner_id, celery_task_id, resource_id, status.

Schema management: app/db/init_db.py provides create_all() (dev/Docker/test boot) and init_database() (also seeds a bootstrap admin in development when DEFAULT_ADMIN_EMAIL/DEFAULT_ADMIN_PASSWORD are set). Alembic is declared for production migrations.

Security model

Password hashing (core/security.py) — bcrypt directly (no passlib), 12 rounds. Passwords longer than bcrypt's 72-byte limit are SHA-256 pre-hashed first. hash_password() enforces a minimum length of 8; verify_password() is constant-time.

JWT — PyJWT, HS256, signed with SECRET_KEY. Access and refresh tokens carry sub, type, iss, iat, nbf, exp, jti and scopes. Decoding verifies signature, issuer and the required claims, and rejects a token whose type does not match what the caller expects. Lifetimes: 30 minutes (access) / 7 days (refresh). AuthService.refresh() rotates the pair and re-checks that the subject still exists and is active. Logout is stateless (tokens are not revocable without a denylist); the client discards them.

Authorisation — get_current_user resolves the bearer token to a User, rejecting missing/malformed tokens, unknown subjects, and inactive accounts. get_optional_user allows anonymous callers (used by rate limiting). require_admin guards admin-only routes.

Tenant isolation — every retrieval is scoped to the caller's user_id via a Chroma where clause, and every document/conversation/task query is filtered by owner_id. Users can only ever see their own data.

Upload policy — extension allow-list, streamed size ceiling (MAX_UPLOAD_SIZE_MB), non-empty content, and normalised extensions.

Transport/response hardening — SecurityHeadersMiddleware sets X-Content-Type-Options: nosniff, X-Frame-Options: DENY, Referrer-Policy: no-referrer. CORS is restricted to CORS_ORIGINS with credentials enabled.

Error privacy — the login path returns one identical error for unknown-email and wrong-password (no account enumeration). Database errors are logged server-side but surface only as a generic storage_error.

Rate limiting

A from-scratch sliding-window-log limiter (core/rate_limit.py) — no slowapi/limits dependency. Each request is recorded in a Redis sorted set keyed ratelimit:{scope}:{identity} (score = timestamp in µs); counting members inside the window yields an exact sliding window. The read/prune/count/write sequence runs in a single atomic MULTI/EXEC pipeline so concurrent workers cannot over-admit. When Redis is unconfigured or unreachable, a thread-safe in-memory store is used instead (keeps tests infrastructure-free and makes a Redis blip degrade rate limiting rather than fail requests).

  • Scopes — default (60/min), auth (10/min), chat (20/min), upload (10/min), all configurable; window is RATE_LIMIT_WINDOW_SECONDS.
  • Identity — user:<id> when authenticated, else ip:<client-ip> (X-Forwarded-For → X-Real-IP → socket peer).
  • Enforcement — dependencies in api/rate_limit_dep.py consume a unit and stash the RateLimitResult on request.state; the middleware copies it to the response as X-RateLimit-*. On breach, a RateLimitError → 429 with Retry-After and X-RateLimit-* headers.
  • Health endpoints are rate-limited too, guaranteeing X-RateLimit-* on every response the API returns.

Background tasks (Celery)

workers/celery_app.py configures Celery with Redis as broker/backend, JSON serialisation, task_track_started, acks_late, reject_on_worker_lost, prefetch=1, max_tasks_per_child=50, UTC timezone and a 20m soft / 25m hard time limit. Queues: default, ingestion (ingestion tasks) and maintenance. celery_app.set_default() registers it as the process-wide default so shared_task resolves correctly from any thread (TestClient, threaded servers).

Tasks (workers/tasks/):

  • ingestion.ingest_document(document_id, task_record_id?) — the full RAG ingestion run; marks the document and task record processing/running, writes progress, and finishes with ready/succeeded or failed. Retries up to 3× with a 15s delay. Owns its own DB session(s) via session_scope().
  • maintenance.reindex_document(document_id, task_record_id?) — resets a document to pending and re-runs ingestion.
  • maintenance.cleanup_stale_tasks(max_age_hours=24) — fails task rows stuck in queued/running past the cutoff.
  • maintenance.health_check() — trivial round-trip probe reporting LLM and embedding health.

Durability — Celery results live in Redis (a cache), so every dispatch is also recorded as a BackgroundTask row. TaskService.get_task() refreshes the durable row from Celery state while a task is in-flight and never commits stale state.

Run a worker from the same image:

celery -A app.workers.celery_app:celery_app worker --loglevel=info

Error envelope

Every failure (application error, validation error, HTTP error, DB error, or an unexpected exception) is serialised by api/errors.py to the identical shape (ErrorResponse):

{
  "code": "email_taken",
  "message": "An account with this email already exists.",
  "details": null,
  "request_id": "3f2a9c…"
}
Handler Status code
AppError (base of the domain hierarchy) per-exception domain code (e.g. email_taken, conversation_not_found, file_too_large)
RequestValidationError 422 validation_error (details = field errors)
StarletteHTTPException as raised http_<status>
SQLAlchemyError 503 storage_error
Exception 500 internal_error

RateLimitError additionally sets Retry-After and X-RateLimit-*. Error details are JSON-coerced with default=str so Pydantic validation contexts (which embed exception objects) never break serialisation.

Domain exceptions (core/exceptions.py): AppError → ValidationError, NotFoundError, ConflictError, AuthenticationError, PermissionDeniedError, RateLimitError, StorageError → VectorStoreError, EmbeddingError, LLMError, DocumentProcessingError, FileValidationError, TaskError.


Frontend documentation

Next.js 16 App Router, React 19, Tailwind CSS v4. The console is a client-rendered SPA shell: "use client" components hold tokens in localStorage and drive the UI; the API remains the authority.

Routing & pages

src/app/ (App Router):

Route File Access Purpose
/ app/page.js public Marketing landing page; redirects authenticated users to /app/chat
/login app/login/page.js public Sign-in form (email + password)
/register app/register/page.js public Account creation (name, email, password with strength checks)
/app app/app/page.js protected Redirects to /app/chat
/app/chat app/app/chat/page.js protected Chat console: SSE-streamed answers + citations
/app/documents app/app/documents/page.js protected Upload, monitor, inspect chunks, edit/reindex/delete
/app/dashboard app/app/dashboard/page.js protected Counters, dependency health, task audit trail
/app/settings app/app/settings/page.js protected Profile, appearance, account controls
(404) app/not-found.js public Global 404
(error) app/app/error.js protected Route error boundary
(loading) app/app/loading.js protected Route-level loading UI

Root layout (app/layout.js) sets metadata, loads Geist fonts, wraps the tree in ThemeProvider → AuthProvider → ToastProvider, and injects a tiny pre-paint script that applies the stored theme to <html> (avoiding a light→dark flash).

Console layout (app/app/layout.js) is the protected shell: it renders the Sidebar + TopBar and an AuthGate that redirects unauthenticated users to /login. Crucially, {children} is always rendered and the gate draws a blocking overlay on top instead of returning early — a layout that returns before rendering {children} would drop every segment in its subtree (Next.js "instant navigation" validation error).

Path alias — @/* → ./src/* (see jsconfig.json).

Authentication flow

src/lib/auth.js exposes AuthProvider + useAuth():

  1. Bootstrap — on mount, read the cached user and token from localStorage. If a token exists, optimistically set the cached user, then call GET /auth/me to confirm (this also exercises token refresh). On failure, clear storage.
  2. login(email, password) — POST /auth/login, store the TokenPair, set user.
  3. register({email, password, full_name}) — POST /auth/register then POST /auth/login (auto sign-in), store tokens.
  4. logout() — call POST /auth/logout (best-effort), clear storage, drop user.
  5. refreshUser() — re-fetch GET /auth/me and update the cached user.

The context value is { user, loading, isAuthenticated, login, register, logout, refreshUser }.

API client

src/lib/api.js is the single backend client.

  • Base URL — NEXT_PUBLIC_API_BASE_URL (default http://localhost:8000) + fixed prefix /api/v1.
  • Token storage (tokenStore) — keys rag.access_token, rag.refresh_token, rag.user in localStorage (SSR-safe guards).
  • request(path, { method, body, headers, auth, retry }) — JSON-encodes non-FormData bodies, attaches Authorization: Bearer <access> when auth, and on a 401 performs one transparent refresh then retries once. On failure it throws an ApiError carrying { status, code, details } parsed from the backend's error envelope. 204 → null; non-JSON → text.
  • refreshAccessToken() — de-duplicates concurrent refreshes via a shared promise; on failure clears the token store.
  • api — thin typed wrappers grouped by domain: auth, users (profile, stats, deactivate), documents (list/stats/get/status/chunks/upload/update/ reindex/delete), chat (ask, conversations CRUD), tasks, health.

SSE client

src/lib/sse.js consumes POST /chat/stream manually via fetch + ReadableStream (the browser EventSource cannot send an Authorization header nor POST).

  • parseEventStream(stream) — an async generator that decodes UTF-8, splits on the blank-line frame boundary, and parses event:/data: lines (: comment lines ignored). Each yielded frame is { event, data } with JSON-parsed data.
  • streamChat(payload, handlers, signal) — POSTs the payload with the bearer token and Accept: text/event-stream, then dispatches frames to onStart / onSources / onToken / onDone / onError. Supports an AbortSignal so the UI can stop generation. Non-OK responses are surfaced as an onError with the backend message.

State providers & theming

  • ThemeProvider (lib/theme.js) — light/dark/system, persisted under rag.theme. applyTheme toggles the dark class on <html>, which the @custom-variant dark (&:where(.dark, .dark *)) rule in globals.css keys off (Tailwind v4 defaults to prefers-color-scheme, which a user control can't override). Follows OS changes while in system mode.
  • ToastProvider (lib/toast.js) — dependency-free { success, error, info, dismiss } notifications rendered as a fixed stack (auto-dismiss timers; errors live longer).
  • Styling (app/globals.css) — @import "tailwindcss", a --brand-* indigo scale exposed via @theme inline, thin scrollbars, and fade-in/slide-in/shimmer keyframes (respecting prefers-reduced-motion).

Component library

  • components/ui.js — presentational primitives: Card, CardHeader, BrandMark, PageHeader, StatCard, Badge, Button, Input, Select, Checkbox, SearchInput, Alert, EmptyState, Spinner, Skeleton, ProgressBar, Modal, CopyButton, StatusBadge, TaskStatusBadge, and formatters formatBytes / formatDate / formatRelative / formatNumber.
  • components/shell.js — console chrome: NAV (Chat, Documents, Dashboard, Settings), Sidebar, TopBar, ThemeSwitcher, UserMenu.
  • components/icons.js — inline SVG icon set (no icon dependency).
  • components/auth-shell.js — the two-column auth layout (brand panel + form card) used by login/register.

Page highlights

  • Chat — conversation rail + transcript; live tokens via streamChat; source chips (cited ones highlighted) with a Stop button; scope selector and "include history" toggle.
  • Documents — drag-and-drop UploadZone, status filters, search, pagination, a ChunkViewer modal, and edit/reindex/delete actions with confirmation.
  • Dashboard — stat tiles, a "Service health" panel (per-dependency readiness), a "Background tasks" list with progress bars, and a worker health probe button.
  • Settings — profile form (PATCH /users/me), theme switcher, account lifecycle (sign out, deactivate), and an "about this deployment" card.

Getting started

Prerequisites

  • Python 3.11
  • Node.js 22 (frontend)
  • PostgreSQL (required — there is no SQLite fallback in the app)
  • Redis (cache, rate limits, Celery broker)
  • ~2 GB disk for the model + embedding weights (downloaded on first use)

1. Backend

cd backend

# (recommended) create a virtual environment
python -m venv .venv && source .venv/bin/activate

# CPU-only PyTorch first (the default index pulls CUDA wheels)
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install -r requirements.txt

# review/point at your services (PostgreSQL + Redis)
#   DATABASE_URL, REDIS_URL, SECRET_KEY, CORS_ORIGINS
$EDITOR .env

# run the API
uvicorn app.main:app --reload --port 8000

The schema is created automatically on startup (init_database). Tables are also created under PostgreSQL first boot.

Offline / quick start. To run without downloading any model weights, set LLM_PROVIDER=stub and EMBEDDING_PROVIDER=hash in .env. The stub answers extractively from retrieved context (fine for smoke tests, not real answers).

2. Frontend

cd frontend
npm install
NEXT_PUBLIC_API_BASE_URL=http://localhost:8000 npm run dev

Open http://localhost:3000, register an account, upload a document, and chat.

The frontend origin must be present in the backend's CORS_ORIGINS (defaults already include http://localhost:3000).

3. Celery worker (optional for ingestion)

With CELERY_TASK_ALWAYS_EAGER=true (the default) ingestion runs inside the API process — no worker is required. To run a real worker instead, set CELERY_TASK_ALWAYS_EAGER=false and start it from the same environment:

cd backend
celery -A app.workers.celery_app:celery_app worker --loglevel=info

Development & testing

Backend tests

The suite is hermetic — no PostgreSQL, no Redis, no model downloads required:

cd backend
pytest                      # full run
pytest tests/unit           # unit only
pytest tests/integration    # integration (API) only
pytest --cov=app            # with coverage

tests/conftest.py configures the environment before importing the app and swaps in: SQLite (temp file, NullPool), in-memory cache, eager Celery, stub LLM + hash embeddings, and an embedded Chroma store in a temp directory. Test layout:

  • tests/unit/ — security, rate limiting, chunking/loading, prompt/chain, vector store/retriever, ingestion pipeline, LLM/embeddings.
  • tests/integration/ — auth, documents, chat (incl. SSE), tasks, health + cross-cutting middleware/OpenAPI.

Backend conventions

  • Layered: api → services → repositories/models, with rag/, llm/, db/, workers/ as infrastructure. Services own transactions; repositories never commit.
  • Singletons (get_* / reset_*) are used for heavy process-wide resources and are reset between tests.
  • All external calls are behind a provider interface with a deterministic test double.

Frontend

cd frontend
npm run dev      # dev server (Turbopack)
npm run build    # production build (standalone output)
npm start        # serve the production build

Config: next.config.mjs enables output: "standalone", React strict mode, cacheComponents, partialPrefetching, and the Tailwind v4 Turbopack loader.


Deployment

Both services ship with multi-stage Dockerfiles; the backend image runs both the API and the Celery worker so they share one pinned dependency set. There is no docker-compose.yml in the repository, so you provide the orchestration (compose, Kubernetes, etc.) and the PostgreSQL + Redis services.

Backend image (backend/Dockerfile)

  • Builder — python:3.11-slim, installs the build toolchain, a venv, CPU-only PyTorch and requirements.txt, then optionally pre-downloads the embedding and LLM weights (PREFETCH_MODELS=true by default) into /opt/hf.
  • Runtime — slim image with only the venv + HF cache + app, runs as a non-root user (appuser), tini as PID 1. Data dirs: /data/chroma, /data/uploads, /data/hf. HEALTHCHECK hits /api/v1/health.
  • Default CMD — Gunicorn + UvicornWorker, 2 workers, --timeout 900 (covers worst-case CPU generation and the SSE keep-alive).
  • Worker — override the command: celery -A app.workers.celery_app:celery_app worker --loglevel=info.

Build args/environment: PREFETCH_MODELS, LLM_MODEL_ID, EMBEDDING_MODEL.

Frontend image (frontend/Dockerfile)

  • Multi-stage node:22-alpine: deps (npm ci) → build (npm run build) → runner serving the standalone output (node server.js) as a non-root user.
  • NEXT_PUBLIC_API_BASE_URL is baked at build time (browser-visible) — pass the public backend URL as a build arg. Listens on 0.0.0.0:3000; HEALTHCHECK fetches /.

Production checklist

  • Set a strong SECRET_KEY (python -c "import secrets; print(secrets.token_urlsafe(64))").
  • Set APP_ENV=production.
  • Restrict CORS_ORIGINS to your real frontend origin(s).
  • Use managed PostgreSQL and Redis; tune DB_POOL_* and REDIS_URL.
  • Serve over TLS (terminate at a reverse proxy; keep X-Accel-Buffering: no so SSE is not buffered).
  • Allocate enough CPU/RAM for local inference (the 0.5B model loads in fp32 by default; use LLM_TORCH_DTYPE=float16 to halve memory).
  • Decide on worker topology: single process (CELERY_TASK_ALWAYS_EAGER=true) vs. separate worker + shared Chroma persistence.
  • Apply Alembic migrations (or create_all()) against the production DB.
  • Persist /data (Chroma + uploads + HF cache) on durable volumes.

License

No license file is included in this repository. Add one (e.g. MIT/Apache-2.0) or set an explicit license before distributing.

About

Upload your own documents (PDF, DOCX, Markdown, CSV, JSON, HTML, TXT), and ask questions in natural language. Answers are generated by a local language model, grounded in the retrieved chunks from your documents, and returned with citations back to the exact source passages — everything runs on your own infrastructure, with no third-party API calls

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages