Skip to content
Arunkumar06-cmdPublic

About

Multimodal document RAG: EVIE/ColPali late-interaction retrieval with an autonomous planner-retriever-verifier agent graph. Demo mode needs no GPU.

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

EVIE RAG

CI CodeQL License: MIT Python FastAPI React

Multimodal document RAG MVP: EVIE/ColPali-style late-interaction retrieval plus an autonomous planner → retriever → verifier agent graph. Demo mode runs without GPU or model downloads; live mode targets a GPU host with Milvus.

GitHub hosts code and CI. A separate GPU provider runs inference. CI never pulls model weights. The GPU image is built in GitHub Actions, pushed to GHCR, then pulled and run on the GPU host over SSH with Docker Compose.

Documentation

Document Contents
docs/ARCHITECTURE.md Request path, agent graph, retriever backends, Milvus token schema, ports.
docs/API.md Every endpoint, request and response shape, status codes, error format.
docs/DEPLOYMENT.md GitHub secrets, gpu-prod environment, first-time indexing, rollback, troubleshooting.
docs/DATASET.md Corpus source, layout, manifest, and indexing workflow.
CONTRIBUTING.md Setup, ground rules, pull request process.
SECURITY.md Private reporting, supported versions, operator hardening checklist.
CHANGELOG.md Release history.

Architecture

React/Vite dashboard -> FastAPI (/api/query) -> agent graph -> retriever -> citations
                                              planner -> retriever -> verifier/answerer
Retrievers: DemoRetriever | Byaldi/ColPali | EVIE + Milvus
Structured output: Pydantic validation; DSPy and Outlines are optional only

Backend lives in backend/app. Frontend lives in frontend. Dataset tooling lives in scripts and dataset.

Features

  • Late-interaction retrieval over page images with EVIE/ColPali, backed by Milvus token vectors and MaxSim aggregation.
  • Autonomous agent graph: planner, retriever, verifier, with a deterministic fallback so demo mode has no optional dependency.
  • Evidence-gated answers. Nothing is marked verified unless a retrieved page clears EVIDENCE_MIN_SCORE.
  • Two interchangeable backends: a CPU demo retriever with no model download, and live byaldi or evie-milvus retrieval.
  • React and Vite dashboard served by FastAPI from a single container and port.
  • Dataset tooling that fetches only the small benchmark subset and never vendors data or weights in git.
  • CI that runs tests, lint, typecheck, and the frontend build with no secrets and no model downloads, plus a gated GPU deployment workflow.

Local setup

python -m venv .venv && source .venv/bin/activate
pip install -e ".[test,lint]"
cp .env.example .env
uvicorn backend.app.main:app --reload --port 8000

Frontend:

cd frontend
npm install
npm run build
npm run dev

Set VITE_API_URL=http://localhost:8000 when the UI and API are on different hosts.

Or use the Makefile:

make help
make install
make check
make demo

Dataset preparation

The corpus is not vendored in git. Use the existing local checkout:

python scripts/prepare_dataset.py --out dataset/manifest.json

Source default: /Users/arunkumar/data/enginuity-bench/task-2-question-answering/qa_dataset.json. Schema: dataset/manifest_schema.json. To fetch only the small Task 2 subset on a fresh host, run python scripts/prepare_dataset.py --download --download-root data/enginuity-bench; this excludes model weights and the large benchmark data.

For a GPU deployment, first validate the page list without loading a model, then index unique figure pages into the Milvus token collection:

python scripts/index_documents.py --manifest dataset/manifest.json --data-root /path/to/task-2-question-answering --dry-run
python scripts/index_documents.py --manifest dataset/manifest.json --data-root /path/to/task-2-question-answering --model-id tencent/EVIE-Preview-4.5B

The indexer stores one FLOAT_VECTOR row per page token and applies MaxSim aggregation during search. The API does not silently switch to demo retrieval when live dependencies are missing.

Docker images

Dockerfile and Dockerfile.demo are lightweight CPU demo images based on python:3.12-slim; they build the React dashboard into /srv/evie-rag/static and serve it from FastAPI. Dockerfile.gpu is the production inference image based on nvidia/cuda:12.1.1-cudnn8-runtime-ubuntu22.04 with Python 3.12 from deadsnakes. Install strategy is PyTorch CUDA 12.1 wheels plus transformers, accelerate, safetensors, then the package retrieval and agents extras which provide byaldi, colpali-engine, pymilvus, langgraph, dspy-ai, and outlines. GPU model download and warmup happen only on the GPU host, never in CI.

Local CPU demo:

docker compose -f docker-compose.demo.yml up -d --build
docker compose -f docker-compose.demo.yml ps
docker compose -f docker-compose.demo.yml down

Production equivalent on a CUDA host:

cp .env.example .env
API_DOCKERFILE=Dockerfile.gpu IMAGE_REF=evie-rag:gpu-local ENV_FILE=.env docker compose up -d --build
docker compose ps

Remote GPU deployment uses the prebuilt GHCR image and does not build locally. IMAGE_REF selects the image, API_DOCKERFILE selects the Dockerfile for local builds, and the protected env file is passed to Compose with --env-file. Set DATA_ROOT_HOST on the GPU host to the directory containing the Task 2 corpus; it is mounted read-only at /data in the API container.

Testing

make check          # ruff, ruff format --check, mypy, pytest
make test           # pytest only

The suite runs with DEMO_MODE=true and HF_HUB_OFFLINE=1, so it never downloads model weights and never needs a GPU. It covers health and readiness, the query response contract, evidence gating, settings parsing, backend selection, traversal-safe page images, bounded ingestion, and the fallback agent graph. CI runs the same commands plus the frontend build and a compose validation, and CodeQL scans Python and TypeScript.

Environment variables

Name Purpose Default
DEMO_MODE true uses deterministic local retriever and fallback graph true in .env.example, false in compose
RETRIEVAL_BACKEND Explicit live backend: byaldi or evie-milvus; ignored in demo mode demo in .env.example, evie-milvus in compose
DATA_ROOT Container path to the Task 2 QA directory /data/task-2-question-answering in compose
DATA_ROOT_HOST Host directory mounted at /data for compose ./data
INDEX_PATH Local late-interaction index path /srv/index/.evie-index in compose
EVIE_MODEL_ID ColPali/EVIE model id for live retrieval tencent/EVIE-Preview-4.5B
EVIE_DIM Native token-vector dimension; must match the index and model 128
HF_TOKEN Hugging Face token for gated models, env only empty
MILVUS_URI Milvus endpoint for live retrieval http://milvus:19530 in compose
MILVUS_COLLECTION Milvus collection for page vectors evie_pages
MINIO_ROOT_USER MinIO access key, required, no default in git placeholder in .env.example
MINIO_ROOT_PASSWORD MinIO secret key, required, no default in git placeholder in .env.example
API_DOCKERFILE Compose Dockerfile selector, Dockerfile for demo or Dockerfile.gpu for production Dockerfile.gpu
API_BIND_ADDRESS Host interface for the API port; keep loopback behind a reverse proxy in production 127.0.0.1
IMAGE_REF Compose image selector, GHCR ref in production evie-rag:local
ENV_FILE Protected env file path used by the deployment command and local Compose invocation .env
CORS_ORIGINS Allowed browser origins http://localhost:5173
TOP_K_DEFAULT Default passages per query 5
EVIDENCE_MIN_SCORE Minimum score before an answer is marked verified 0.05
MAX_UPLOAD_MB Upload size cap 50
VITE_API_URL Frontend API base URL http://localhost:8000

See .env.example for a copy-paste starting point with no secret values. Compose fails at startup when MINIO_ROOT_USER or MINIO_ROOT_PASSWORD are missing. Use real secret values only in .env locally or in the protected host file referenced by DEPLOY_ENV_FILE.

Model token handling

HF_TOKEN and EVIE_MODEL_ID stay out of git, images, and logs. Set HF_TOKEN only in the local .env for testing gated models or in the GPU host file referenced by DEPLOY_ENV_FILE for production. Leave both empty for public models and for demo mode. CI exports HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1 with DEMO_MODE=true so no weights are downloaded during tests, lint, typecheck, or the frontend build.

GitHub secrets

CI needs no secrets. Deployment uses only these names:

  • GITHUB_TOKEN (automatic, for GHCR push; no personal token required)
  • GPU_HOST
  • GPU_USER
  • GPU_SSH_KEY
  • DEPLOY_PATH
  • DEPLOY_ENV_FILE (path on the GPU host to a protected env file)
  • GHCR_READ_TOKEN (only needed when the GitHub package is private; a read-only token is sent to the host over the SSH session using a temporary Docker config that is removed after deployment)
  • GHCR_USERNAME (optional; defaults to the GitHub actor when using a private package)

Deployment flow

  1. Tag v* or run deploy manually with approval for the gpu-prod environment.
  2. build-push builds Dockerfile.gpu and pushes tag plus commit-SHA tags to GHCR with GITHUB_TOKEN.
  3. deploy-gpu opens one SSH session, writes the key with 0600 and a trap-based cleanup, validates a tag-safe lowercase sha-<commit> image ref, validates the host compose file, pulls that ref and its dependencies, and refreshes the deployment with the protected env file.
  4. On the GPU host, DEPLOY_PATH holds docker-compose.yml; DEPLOY_ENV_FILE supplies HF_TOKEN, MINIO_ROOT_USER, MINIO_ROOT_PASSWORD, and live-mode values outside git. Compose loads that file through --env-file and ENV_FILE without sourcing it or echoing secrets.

The workflow supports any SSH-reachable CUDA host with Docker, the NVIDIA container runtime, and a checkout directory. The API binds to 127.0.0.1 by default; expose it through an authenticated TLS reverse proxy rather than publishing the port directly. No cloud-specific actions, provider CLI, or long-lived credentials beyond the existing secrets are required.

GPU sizing caveat

A 4.5B late-interaction vision model needs a CUDA GPU host (24GB+ VRAM recommended, 16GB minimum depending on quantization and batch size). The host needs an NVIDIA driver, nvidia-container-toolkit, Docker Compose v2.24 or newer for env_file path mapping, and persistent volumes for evie-index, milvus-data, minio-data, and etcd-data. CPU-only CI never loads model weights; it exercises demo retrieval, the fallback graph, and the frontend build only. GPU image pulls and model warmup happen exclusively on the GPU host.

Security notes

  • No secrets in git; .env, .env.*, *.pem, *.key, and .image-ref are ignored and only .env.example is committed.
  • HF_TOKEN, MINIO_ROOT_USER, and MINIO_ROOT_PASSWORD travel via environment or DEPLOY_ENV_FILE, never in logs or images.
  • Uploads are size-capped and indexed as text placeholders until OCR/embeddings run.
  • Error handlers return generic messages plus request IDs; tracebacks never leak to clients.
  • CORS is allowlisted; SSH uses strict host-key acceptance and a one-shot key file.

Report vulnerabilities privately as described in SECURITY.md.

Project layout

backend/app/       FastAPI service, settings, retriever, agent graph, store
backend/tests/     Pytest suite that runs without a GPU or model weights
frontend/          React and Vite dashboard
scripts/           Dataset preparation and Milvus indexing
dataset/           Manifest schema and dataset notes
docs/              Architecture, API, deployment, and dataset documentation
Dockerfile         CPU demo image
Dockerfile.demo    CPU demo image with an explicit name
Dockerfile.gpu     CUDA production image
docker-compose.yml         Production stack: API, Milvus, etcd, MinIO
docker-compose.demo.yml    CPU-only demo stack
.github/workflows/          CI, CodeQL, and the gated GPU deployment

Contributing

Issues and pull requests are welcome. Start with CONTRIBUTING.md for setup, ground rules, and the review checklist. Security problems go through SECURITY.md instead of a public issue.

License

Released under the MIT License.

About

Multimodal document RAG: EVIE/ColPali late-interaction retrieval with an autonomous planner-retriever-verifier agent graph. Demo mode needs no GPU.

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages