简体中文

Data Advisor

A self-hosted file and data analysis agent for teams

Data Advisor combines regular chat, private-file Q&A, shared project knowledge, and isolated data analysis in one recoverable conversation workspace. It is built for teams that need self-hosting, authorization, pinned file versions, traceable citations, and multiple model providers.

Data Advisor is intentionally a simple, extensible skeleton for teams to adopt, customize, and evolve into their own product.

What it does

Capability	Behavior
Regular chat	Start without uploading files and keep durable context in a Thread.
Project knowledge	Link multiple Projects to a Thread; current files are indexed and retrieved automatically.
Private documents	PDF, DOCX, PNG, and JPEG remain scoped to the current chat unless explicitly added to a Project.
Data analysis	Run deterministic CSV/XLSX computation in an isolated Sandbox and produce downloadable artifacts.
Reliable execution	Persist Runs with streaming output, cancellation, retry, and refresh-safe state recovery.
Multiple models	Connect DeepSeek, Doubao, Anthropic, and OpenAI through one Registry; enable models independently in Admin.

Screenshots

Model configuration	Operations

Quick start

Docker Desktop and Docker Compose are required. Python 3.13, uv, Node.js 20.19+, and pnpm 10 are needed only for host-process development.

1. Copy the environment template:

   cp .env.example .env

   On PowerShell, use Copy-Item .env.example .env.

2. Set independent values for at least:

   • POSTGRES_PASSWORD
   • MINIO_ROOT_PASSWORD
   • JWT_SECRET
   • OPENVIKING_API_KEY
   • SANDBOX_INTERNAL_TOKEN

   Configure Chat providers as needed:

   Environment variable	Models made available
   DEEPSEEK_API_KEY	deepseek-v4-flash, deepseek-v4-pro
   VOLCENGINE_API_KEY	doubao-seed-2-0-lite-260428; also used for OpenViking project-knowledge processing
   ANTHROPIC_API_KEY	Claude Fable, Opus, Sonnet, Haiku
   OPENAI_API_KEY	gpt-5.5, gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna

   Never commit .env or inject provider keys into the browser or Sandbox.

   Important: configure OpenViking models separately. OpenViking is not controlled by the Admin model UI; that page selects user Chat models only. Set OPENVIKING_VLM_MODEL and OPENVIKING_EMBEDDING_MODEL in .env (template: .env.example); the actual model wiring is under services.openviking.command in docker-compose.yml.

3. Start the full environment:

   docker compose up --build --wait
   docker compose ps

4. Open the product UI: http://127.0.0.1:3000

After the first deployment, promote an existing user to system administrator:

uv run data-advisor admin promote --email admin@example.com

Common endpoints:

• Product UI: http://127.0.0.1:3000
• API documentation: http://127.0.0.1:8000/docs
• Admin operations: http://127.0.0.1:3000/admin/operations
• Admin models: http://127.0.0.1:3000/admin/models
• MinIO Console: http://127.0.0.1:9001
• OpenViking Studio: http://127.0.0.1:1933/studio

Core concepts

Concept	Meaning
Project	An administrator-created shared knowledge collection containing zero or more files and versions.
FileVersion	A pinned file revision. Re-uploading or changing a file creates a new version; existing chats do not switch automatically.
Thread	A durable user conversation that can link zero or more Projects and pin chat attachments.
Run	One execution inside a Thread with fixed Chat model, Tools, Skills, budget, and input versions.
Message	One user or assistant message that may link Citations, Artifacts, and Run state.

Files follow two explicit paths:

• Project knowledge: OpenViking indexes the current files in a Project. A linked Thread retrieves those sources automatically without requiring per-file selection.
• Chat attachments: temporary PDF, DOCX, PNG, JPEG, CSV, and XLSX inputs pin exact FileVersions and are not written to OpenViking. Publish them to a Project explicitly when they should be shared.

Architecture

flowchart LR
    Web["Web / Gateway"] --> API["FastAPI API"]

    API -->|"Identity, authorization, Message, Run, Outbox"| PG[("PostgreSQL")]
    API -->|"Upload files"| MinIO[("MinIO")]

    Dispatcher["Run Dispatcher<br/>reliable delivery and reconciliation"] -->|"Claim Run Outbox"| PG
    Dispatcher -->|"Enqueue ARQ job"| Redis[("Redis / ARQ")]

    RunWorker["Run Worker<br/>Agent execution"] -->|"Consume job"| Redis
    RunWorker <-->|"Read Run / write answer, Citation, Artifact"| PG
    RunWorker -->|"Read pinned FileVersion"| MinIO
    RunWorker --> LLM["Run-selected Chat model"]
    RunWorker --> Provisioner["Sandbox Provisioner"]
    Provisioner --> Sandbox["Isolated Sandbox container"]
    RunWorker -->|"Authorized recall"| OV[("OpenViking")]

    Indexer["Project Resource Indexer"] -->|"Claim knowledge Outbox"| PG
    Indexer -->|"Read project files"| MinIO
    Indexer -->|"Synchronize shared project knowledge"| OV

    Memory["Conversation Memory Worker"] -->|"Claim memory Outbox"| PG
    Memory -->|"Capture / Commit summaries and preferences"| OV

    OV --> OVModels["Deployment-configured VLM / Embedding"]

    Web -->|"Admin operations"| API
    API -->|"Aggregate heartbeats, backlog, and leases"| PG

Loading

A Chat Run follows this path:

1. The API revalidates the user, Thread, Projects, and attachments server-side, then writes the Message, Run, and Run Outbox in one PostgreSQL transaction.
2. The Run Dispatcher claims the Outbox record and submits it to Redis/ARQ. Durable database intent keeps a temporary Redis outage retryable and reconcilable.
3. The Run Worker pins the model and inputs, then invokes project retrieval, private-document Tools, or an isolated Sandbox when required.
4. Answers, Citations, and Artifacts commit to PostgreSQL before Redis/SSE projects them to the browser. A disconnected browser recovers from database state.

Background services

Compose service	Responsibility
run-dispatcher	Reliably delivers Runs, reconciles stuck or expired work, and triggers Sandbox cleanup; never executes an LLM or Tool.
run-worker	Executes Agents, document Tools, and Sandbox Tools; reaches approved Chat providers through model-egress.
project-resource-indexer	Synchronizes each Project's current MinIO files into OpenViking project knowledge.
conversation-memory-worker	Synchronizes committed final user/assistant messages and creates cross-Run summaries and preferences; never receives Tool traces or drafts.

The Admin operations page aggregates PostgreSQL service heartbeats, backlog, leases, and 24-hour compression metering. /api/system/health remains a lightweight liveness endpoint.

OpenViking is the current implementation for project knowledge and cross-Run conversation memory. To replace it, implement the ContextBackend and ContextSessionBackend ports and register the new provider in the Worker Runtime.

Key boundaries

• PostgreSQL remains authoritative for identity, authorization, original Messages, and execution state; every Run and Tool access is reauthorized.
• The browser selects only server-approved, Admin-enabled Chat models and cannot modify capabilities, provider addresses, or credentials.
• The Run-selected capable Chat model reads PDFs and images; DOCX text is extracted server-side.
• Private document context is prepared once per FileVersion and stored as a Markdown derivative beside the MinIO source. PostgreSQL stores authorization metadata, and later questions reuse it.
• CSV/XLSX enters a Thread-scoped logical Sandbox only through controlled Tools. Only sandbox-provisioner can access the Docker Socket.
• Deep Runs enable one layer of specialist sub-agents only after explicit selection; formal file computation still uses the Sandbox.
• OpenViking receives only committed final user/assistant Messages, never Tool traces, Todos, sub-agent drafts, or StateBackend files.

The static prompt lives at packages/runtime/src/agent_runtime/prompt.py. Runtime Context assembles dynamic identity, authorization, file content, and retrieval results per Run. Versioned built-in Skills live in skills/.

Repository layout

apps/
  api/                    FastAPI API and dependency composition
  worker/                 background dispatch, execution, and synchronization
  web/                    Next.js Web UI
packages/
  core/                   business rules, authorization, persistence, and files
  runtime/                Agent orchestration, Context, Tools, Skills, and Sandbox contracts
  llm/                    model catalog and provider factories
sandbox-provisioner/      only internal service allowed to access the Docker Socket
skills/                   versioned, read-only built-in Skills
migrations/               Alembic database migrations
infra/                    Docker images and Nginx configuration
tests/                    unit, integration, and regression tests
docs/                     specs, plans, ADRs, and README images

Dependencies flow from api/worker -> core/runtime/llm. Core does not depend on Apps, and neither the API nor workers access the Docker Socket.

Development

Common Docker commands

Rebuild affected services after code changes:

docker compose up -d --build --wait api run-dispatcher run-worker project-resource-indexer conversation-memory-worker
docker compose up -d --build --wait web gateway

Apply database migrations:

docker compose up -d --wait postgres
docker compose build migrate
docker compose run --rm migrate

Follow background logs:

docker compose logs -f api run-dispatcher run-worker project-resource-indexer conversation-memory-worker

Stop services while retaining PostgreSQL/MinIO named volumes:

docker compose down

Host-process development

Run infrastructure in Docker and the API/Web on the host:

uv sync --locked --python 3.13
uv run alembic upgrade head
uv run uvicorn api_server.main:app --host 127.0.0.1 --port 8000

In another terminal:

pnpm --dir apps/web install --frozen-lockfile
pnpm --dir apps/web dev

Tests

Default tests use Fakes and Mocks and never call a real Embedding or paid model:

uv run pytest
pnpm --dir apps/web test

Paid-provider and local Sandbox smoke tests run only behind explicit switches:

$env:RUN_DEEPSEEK_SMOKE_TEST = "1"
uv run pytest tests/integration/test_deepseek_smoke.py

$env:RUN_DOUBAO_SMOKE_TEST = "1"
uv run pytest tests/integration/test_doubao_smoke.py

$env:RUN_TABULAR_SANDBOX_SMOKE = "1"
uv run pytest tests/integration/test_tabular_sandbox_smoke.py

Run the complete verification suite before submitting changes:

uv sync --locked --python 3.13
uv run pytest
uv run ruff check .
uv run ruff format --check .
uv run alembic upgrade head --sql
pnpm --dir apps/web install --frozen-lockfile
pnpm --dir apps/web test
pnpm --dir apps/web lint
pnpm --dir apps/web format:check
pnpm --dir apps/web typecheck
pnpm --dir apps/web build
docker compose config --quiet

Contributing

Issues and pull requests are welcome. Add tests for behavior changes, and keep default verification free of real Embedding or paid-model calls.

License

Data Advisor is released under the MIT License.