DataScope is a workspace-based exploratory data analysis app for CSV datasets. It helps users upload data, inspect schema and quality, run deterministic EDA tools, generate charts, and ask guided questions about the dataset without turning the LLM into the source of truth.
The core idea is simple: compute statistics and visualizations with reliable tooling, then use an agent layer to plan analysis steps and explain the structured results in plain language.
- Organizes analysis around user workspaces and dataset versions.
- Accepts CSV uploads with validation, profiling, and durable storage.
- Runs deterministic EDA tools for overview metrics, column summaries, relationships, plotting, and data-quality checks.
- Persists analysis runs, tool executions, generated artifacts, and conversation history so work can be resumed.
- Streams live job status to the frontend while keeping Postgres as the recovery source.
- Uses LLM providers for planning and interpretation, while keeping raw dataset access bounded and tool-driven.
src/frontend/ Next.js app for login, workspaces, upload, analysis, charts
src/backend/ FastAPI API, Celery workers, EDA tools, agent orchestration
assets/ Architecture and design images
docs/ Product, API, schema, system, frontend, and agent designs
docker-compose.yml Local Postgres, Redis, and backend runtime
At a high level, the system is split into:
| Component | Role |
|---|---|
| Frontend | Next.js UI for workspaces, dataset review, chat, tool selection, and results. |
| API | FastAPI service for auth, uploads, analysis requests, history, and WebSockets. |
| Workers | Celery processes for heavier EDA jobs, plots, retries, and background tasks. |
| Database | Postgres stores users, workspaces, datasets, versions, runs, messages, and artifacts. |
| Queue/cache | Redis backs Celery and stores short-lived reusable analysis outputs. |
| Storage | Cloudflare R2 stores uploaded CSV files and generated plot artifacts. |
| Agent layer | Plans valid tool usage and explains computed results using LLM providers. |
The backend has a more detailed README at
src/backend/README.md, including full API routes,
environment variables, worker commands, and backend-specific rules.
Prerequisites: Docker Desktop, uv, Node.js.
- Create
src/backend/.env(see.env.example). - For local runs use:
DATABASE_URL=postgresql+psycopg://datascope:datascope@localhost:5432/datascopeREDIS_URL=redis://localhost:6379/0
- Add the R2, Groq, Gemini and Google OAuth values.
- No spaces around
=and no quotes around values.
Run from the repository root:
docker compose up -d postgres redis- This pulls the images on first run, creates the containers and keeps them running in the background.
- Always name the services. A bare
docker compose upalso starts the composebackendservice, which clashes with port 8000.
Check they are running (postgres should show healthy):
docker compose psView logs when needed:
docker compose logs -f postgres rediscd src/backend
uv sync
uv run alembic upgrade head
uv run uvicorn app.main:app --reloadalembic upgrade headcreates the database tables. Run it on first setup and after pulling new migrations.- API docs: http://localhost:8000/docs
- Health check: http://localhost:8000/health
In a new terminal:
cd src/backend
uv run celery -A app.workers.celery_app:celery_app worker --loglevel=info --pool=soloIn a new terminal:
cd src/frontend
npm install
npm run devStop the containers and keep the data:
docker compose stop postgres redisRemove the containers and keep the data volumes:
docker compose downRemove the containers and delete all database data:
docker compose down -vcd C:\DataScope\src\backend
uv run pytestThe tests cover workspace routes, dataset upload/validation, storage behavior, analysis routes/tasks, deterministic EDA tools, conversations, and agent orchestration.
The current implementation is focused on CSV-first EDA:
- Google OAuth based authentication.
- Workspace and dataset lifecycle APIs.
- Immutable dataset versions for uploaded files.
- Deterministic analysis tools before LLM explanation.
- Live progress updates over WebSockets.
- Tests for routes, services, tools, workers, and orchestration behavior.
Future work can expand this into richer dataset connectors, scheduled analyses, team collaboration, stronger artifact management, and deeper evaluation suites for agent behavior.
A practical production deployment can keep the app split by responsibility:
| Layer | Plausible option |
|---|---|
| Frontend | Vercel, Netlify, or a containerized Next.js host. |
| API | Containerized FastAPI service on Render, Fly.io, Railway, AWS ECS, GCP Cloud Run, or Kubernetes. |
| Workers | Separate Celery worker process/container scaled independently from the API. |
| Database | Managed Postgres instance via Supabase with automated backups and migration jobs. |
| Redis | Managed Redis for Celery broker, progress events, and hot cache entries. |
| Object storage | Cloudflare R2 bucket for CSV uploads and generated chart files. |
| Secrets | Platform-managed environment variables for OAuth, session, LLM, database, Redis, and R2 credentials. |
Detailed Design docs live in docs/, while implementation lives under src/.The root README is meant to explain the whole platform and deployment shape; service-level
details should stay in the frontend and backend READMEs.