Multimodal document RAG MVP: EVIE/ColPali-style late-interaction retrieval plus an autonomous planner → retriever → verifier agent graph. Demo mode runs without GPU or model downloads; live mode targets a GPU host with Milvus.
GitHub hosts code and CI. A separate GPU provider runs inference. CI never pulls model weights. The GPU image is built in GitHub Actions, pushed to GHCR, then pulled and run on the GPU host over SSH with Docker Compose.
| Document | Contents |
|---|---|
| docs/ARCHITECTURE.md | Request path, agent graph, retriever backends, Milvus token schema, ports. |
| docs/API.md | Every endpoint, request and response shape, status codes, error format. |
| docs/DEPLOYMENT.md | GitHub secrets, gpu-prod environment, first-time indexing, rollback, troubleshooting. |
| docs/DATASET.md | Corpus source, layout, manifest, and indexing workflow. |
| CONTRIBUTING.md | Setup, ground rules, pull request process. |
| SECURITY.md | Private reporting, supported versions, operator hardening checklist. |
| CHANGELOG.md | Release history. |
React/Vite dashboard -> FastAPI (/api/query) -> agent graph -> retriever -> citations
planner -> retriever -> verifier/answerer
Retrievers: DemoRetriever | Byaldi/ColPali | EVIE + Milvus
Structured output: Pydantic validation; DSPy and Outlines are optional only
Backend lives in backend/app. Frontend lives in frontend. Dataset tooling lives in scripts and dataset.
- Late-interaction retrieval over page images with EVIE/ColPali, backed by Milvus token vectors and MaxSim aggregation.
- Autonomous agent graph: planner, retriever, verifier, with a deterministic fallback so demo mode has no optional dependency.
- Evidence-gated answers. Nothing is marked verified unless a retrieved page
clears
EVIDENCE_MIN_SCORE. - Two interchangeable backends: a CPU demo retriever with no model download, and
live
byaldiorevie-milvusretrieval. - React and Vite dashboard served by FastAPI from a single container and port.
- Dataset tooling that fetches only the small benchmark subset and never vendors data or weights in git.
- CI that runs tests, lint, typecheck, and the frontend build with no secrets and no model downloads, plus a gated GPU deployment workflow.
python -m venv .venv && source .venv/bin/activate
pip install -e ".[test,lint]"
cp .env.example .env
uvicorn backend.app.main:app --reload --port 8000Frontend:
cd frontend
npm install
npm run build
npm run devSet VITE_API_URL=http://localhost:8000 when the UI and API are on different hosts.
Or use the Makefile:
make help
make install
make check
make demoThe corpus is not vendored in git. Use the existing local checkout:
python scripts/prepare_dataset.py --out dataset/manifest.jsonSource default: /Users/arunkumar/data/enginuity-bench/task-2-question-answering/qa_dataset.json. Schema: dataset/manifest_schema.json. To fetch only the small Task 2 subset on a fresh host, run python scripts/prepare_dataset.py --download --download-root data/enginuity-bench; this excludes model weights and the large benchmark data.
For a GPU deployment, first validate the page list without loading a model, then index unique figure pages into the Milvus token collection:
python scripts/index_documents.py --manifest dataset/manifest.json --data-root /path/to/task-2-question-answering --dry-run
python scripts/index_documents.py --manifest dataset/manifest.json --data-root /path/to/task-2-question-answering --model-id tencent/EVIE-Preview-4.5BThe indexer stores one FLOAT_VECTOR row per page token and applies MaxSim aggregation during search. The API does not silently switch to demo retrieval when live dependencies are missing.
Dockerfile and Dockerfile.demo are lightweight CPU demo images based on python:3.12-slim; they build the React dashboard into /srv/evie-rag/static and serve it from FastAPI. Dockerfile.gpu is the production inference image based on nvidia/cuda:12.1.1-cudnn8-runtime-ubuntu22.04 with Python 3.12 from deadsnakes. Install strategy is PyTorch CUDA 12.1 wheels plus transformers, accelerate, safetensors, then the package retrieval and agents extras which provide byaldi, colpali-engine, pymilvus, langgraph, dspy-ai, and outlines. GPU model download and warmup happen only on the GPU host, never in CI.
Local CPU demo:
docker compose -f docker-compose.demo.yml up -d --build
docker compose -f docker-compose.demo.yml ps
docker compose -f docker-compose.demo.yml downProduction equivalent on a CUDA host:
cp .env.example .env
API_DOCKERFILE=Dockerfile.gpu IMAGE_REF=evie-rag:gpu-local ENV_FILE=.env docker compose up -d --build
docker compose psRemote GPU deployment uses the prebuilt GHCR image and does not build locally. IMAGE_REF selects the image, API_DOCKERFILE selects the Dockerfile for local builds, and the protected env file is passed to Compose with --env-file. Set DATA_ROOT_HOST on the GPU host to the directory containing the Task 2 corpus; it is mounted read-only at /data in the API container.
make check # ruff, ruff format --check, mypy, pytest
make test # pytest onlyThe suite runs with DEMO_MODE=true and HF_HUB_OFFLINE=1, so it never
downloads model weights and never needs a GPU. It covers health and readiness,
the query response contract, evidence gating, settings parsing, backend
selection, traversal-safe page images, bounded ingestion, and the fallback agent
graph. CI runs the same commands plus the frontend build and a compose
validation, and CodeQL scans Python and TypeScript.
| Name | Purpose | Default |
|---|---|---|
DEMO_MODE |
true uses deterministic local retriever and fallback graph |
true in .env.example, false in compose |
RETRIEVAL_BACKEND |
Explicit live backend: byaldi or evie-milvus; ignored in demo mode |
demo in .env.example, evie-milvus in compose |
DATA_ROOT |
Container path to the Task 2 QA directory | /data/task-2-question-answering in compose |
DATA_ROOT_HOST |
Host directory mounted at /data for compose |
./data |
INDEX_PATH |
Local late-interaction index path | /srv/index/.evie-index in compose |
EVIE_MODEL_ID |
ColPali/EVIE model id for live retrieval | tencent/EVIE-Preview-4.5B |
EVIE_DIM |
Native token-vector dimension; must match the index and model | 128 |
HF_TOKEN |
Hugging Face token for gated models, env only | empty |
MILVUS_URI |
Milvus endpoint for live retrieval | http://milvus:19530 in compose |
MILVUS_COLLECTION |
Milvus collection for page vectors | evie_pages |
MINIO_ROOT_USER |
MinIO access key, required, no default in git | placeholder in .env.example |
MINIO_ROOT_PASSWORD |
MinIO secret key, required, no default in git | placeholder in .env.example |
API_DOCKERFILE |
Compose Dockerfile selector, Dockerfile for demo or Dockerfile.gpu for production |
Dockerfile.gpu |
API_BIND_ADDRESS |
Host interface for the API port; keep loopback behind a reverse proxy in production | 127.0.0.1 |
IMAGE_REF |
Compose image selector, GHCR ref in production | evie-rag:local |
ENV_FILE |
Protected env file path used by the deployment command and local Compose invocation | .env |
CORS_ORIGINS |
Allowed browser origins | http://localhost:5173 |
TOP_K_DEFAULT |
Default passages per query | 5 |
EVIDENCE_MIN_SCORE |
Minimum score before an answer is marked verified | 0.05 |
MAX_UPLOAD_MB |
Upload size cap | 50 |
VITE_API_URL |
Frontend API base URL | http://localhost:8000 |
See .env.example for a copy-paste starting point with no secret values. Compose fails at startup when MINIO_ROOT_USER or MINIO_ROOT_PASSWORD are missing. Use real secret values only in .env locally or in the protected host file referenced by DEPLOY_ENV_FILE.
HF_TOKEN and EVIE_MODEL_ID stay out of git, images, and logs. Set HF_TOKEN only in the local .env for testing gated models or in the GPU host file referenced by DEPLOY_ENV_FILE for production. Leave both empty for public models and for demo mode. CI exports HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1 with DEMO_MODE=true so no weights are downloaded during tests, lint, typecheck, or the frontend build.
CI needs no secrets. Deployment uses only these names:
GITHUB_TOKEN(automatic, for GHCR push; no personal token required)GPU_HOSTGPU_USERGPU_SSH_KEYDEPLOY_PATHDEPLOY_ENV_FILE(path on the GPU host to a protected env file)GHCR_READ_TOKEN(only needed when the GitHub package is private; a read-only token is sent to the host over the SSH session using a temporary Docker config that is removed after deployment)GHCR_USERNAME(optional; defaults to the GitHub actor when using a private package)
- Tag
v*or rundeploymanually with approval for thegpu-prodenvironment. build-pushbuildsDockerfile.gpuand pushes tag plus commit-SHA tags to GHCR withGITHUB_TOKEN.deploy-gpuopens one SSH session, writes the key with0600and a trap-based cleanup, validates a tag-safe lowercasesha-<commit>image ref, validates the host compose file, pulls that ref and its dependencies, and refreshes the deployment with the protected env file.- On the GPU host,
DEPLOY_PATHholdsdocker-compose.yml;DEPLOY_ENV_FILEsuppliesHF_TOKEN,MINIO_ROOT_USER,MINIO_ROOT_PASSWORD, and live-mode values outside git. Compose loads that file through--env-fileandENV_FILEwithout sourcing it or echoing secrets.
The workflow supports any SSH-reachable CUDA host with Docker, the NVIDIA container runtime, and a checkout directory. The API binds to 127.0.0.1 by default; expose it through an authenticated TLS reverse proxy rather than publishing the port directly. No cloud-specific actions, provider CLI, or long-lived credentials beyond the existing secrets are required.
A 4.5B late-interaction vision model needs a CUDA GPU host (24GB+ VRAM recommended, 16GB minimum depending on quantization and batch size). The host needs an NVIDIA driver, nvidia-container-toolkit, Docker Compose v2.24 or newer for env_file path mapping, and persistent volumes for evie-index, milvus-data, minio-data, and etcd-data. CPU-only CI never loads model weights; it exercises demo retrieval, the fallback graph, and the frontend build only. GPU image pulls and model warmup happen exclusively on the GPU host.
- No secrets in git;
.env,.env.*,*.pem,*.key, and.image-refare ignored and only.env.exampleis committed. HF_TOKEN,MINIO_ROOT_USER, andMINIO_ROOT_PASSWORDtravel via environment orDEPLOY_ENV_FILE, never in logs or images.- Uploads are size-capped and indexed as text placeholders until OCR/embeddings run.
- Error handlers return generic messages plus request IDs; tracebacks never leak to clients.
- CORS is allowlisted; SSH uses strict host-key acceptance and a one-shot key file.
Report vulnerabilities privately as described in SECURITY.md.
backend/app/ FastAPI service, settings, retriever, agent graph, store
backend/tests/ Pytest suite that runs without a GPU or model weights
frontend/ React and Vite dashboard
scripts/ Dataset preparation and Milvus indexing
dataset/ Manifest schema and dataset notes
docs/ Architecture, API, deployment, and dataset documentation
Dockerfile CPU demo image
Dockerfile.demo CPU demo image with an explicit name
Dockerfile.gpu CUDA production image
docker-compose.yml Production stack: API, Milvus, etcd, MinIO
docker-compose.demo.yml CPU-only demo stack
.github/workflows/ CI, CodeQL, and the gated GPU deployment
Issues and pull requests are welcome. Start with CONTRIBUTING.md for setup, ground rules, and the review checklist. Security problems go through SECURITY.md instead of a public issue.
Released under the MIT License.