Autonomous software development via iterative evaluation. Inspired by Karpathy's autoresearch — an AI agent that autonomously builds, evaluates, and iterates on software applications using objective quality metrics.
Instead of training a model overnight, AutoDev builds software overnight: write code → start server → evaluate with Playwright + Claude Vision → fix issues → keep or discard → repeat.
┌─────────────────────────────────────────────────────┐
│ program.md │
│ (goal, requirements, weights, MCP) │
└──────────────────────┬──────────────────────────────┘
│
▼
┌─── DECOMPOSE ───┐
│ Goal → Tasks │
│ (Kanban board) │
└────────┬────────┘
│
┌────────────▼────────────┐
│ SELECT NEXT TASK │◄──────────────┐
└────────────┬────────────┘ │
│ │
┌────────────▼────────────┐ │
│ DEVELOP │ │
│ (Claude + tools) │ │
└────────────┬────────────┘ │
│ │
┌────────────▼────────────┐ │
│ EVALUATE │ │
│ Playwright + Vision │ │
└────────────┬────────────┘ │
│ │
┌────────────▼────────────┐ │
│ IMPROVE │ │
│ Fix specific issues │ │
└────────────┬────────────┘ │
│ │
┌────────────▼────────────┐ │
│ RE-EVALUATE │ │
└────────────┬────────────┘ │
│ │
┌────────────▼────────────┐ score > │
│ DECIDE │────best?──────┘
│ keep / revert / block │ repeat
└─────────────────────────┘
Each iteration gets two shots — an initial build and a repair pass driven by concrete Playwright evidence (console errors, failing interactions, visual defects, accessibility violations).
AutoDev ships with a real-time React dashboard for monitoring and controlling the autonomous loop. The GUI connects to the FastAPI backend over REST + WebSocket, giving you live visibility into every iteration.
On first launch, the setup screen guides you through configuring program.md — the single human control surface that tells AutoDev what to build. A Monaco-powered markdown editor lets you define your goal, requirements, evaluation weights, and MCP server configurations. Click Save & Decompose Tasks and Claude breaks your goal into a prioritized task plan.
The primary view for managing the development pipeline. Tasks flow through six columns — Backlog, In Progress, Evaluating, Improving, Done, and Blocked — as the autonomous loop works through them. You can drag-and-drop to reprioritize, add custom tasks, filter by name, and manually unblock stuck tasks. The progress bar at the top shows overall completion. Color-coded priority dots (red = high, orange = medium, blue = low) give quick visual triage. The right panel shows live agent activity and evaluation screenshots as they happen.
Track quality progression across every iteration. The Score History chart plots composite and per-dimension scores over time, with toggles for each evaluation dimension (Functionality, Visual, Performance, Accessibility, Structure, Errors). Green markers indicate kept iterations; red markers show reverted ones. Below the chart, the Iteration Log table provides a detailed breakdown of every iteration — which task was attempted, individual dimension scores, score deltas, and whether the change was kept or discarded.
A split-pane file explorer and source viewer for the software being built. The left panel shows the workspace file tree (auto-refreshed), and clicking any file renders its contents in the right panel. Useful for inspecting what the agent has written without leaving the dashboard.
Edit program.md live without restarting the loop. Changes take effect on the next iteration — the orchestrator re-reads the program file at the start of each cycle. The Monaco editor provides syntax highlighting, line numbers, and word wrap for comfortable editing. Below the editor, the MCP Servers panel shows which tool servers are configured and connected, with tool counts per server.
| Dimension | What it measures |
|---|---|
| Functionality | Playwright interaction testing, console errors |
| Visual | Claude Vision screenshot analysis against quality criteria |
| Performance | window.performance API — load times, DOM size |
| Accessibility | ARIA snapshot, heading structure, alt text, labels |
| Structure | File organization, no giant files, separation of concerns |
| Errors | Console error count penalty |
Weights are configurable in program.md and can be adjusted live while the loop runs.
- Python 3.11+
- Node.js 18+
- An API key for your chosen provider (see Multi-Provider Support below)
pip install -e .
playwright install chromium
cd gui && npm install && cd ..-
Edit
program.md— describe what you want to build -
Start the loop:
python run.py
-
Or with the GUI:
# Terminal 1: API server python api.py # Terminal 2: React dev server cd gui && npm run dev
Open
http://localhost:5174 -
Or as an MCP server (for Claude Code):
python server.py
| Variable | Default | Description |
|---|---|---|
ANTHROPIC_API_KEY |
— | Anthropic API key (if using Claude) |
OPENAI_API_KEY |
— | OpenAI API key (if using GPT models) |
AUTODEV_MODEL |
anthropic/claude-sonnet-4-6 |
Global model override (LiteLLM format) |
AUTODEV_MAX_ITERATIONS |
100 |
Maximum iterations |
AutoDev supports any LLM provider via LiteLLM. Configure the model in program.md using provider/model-name format:
Anthropic (default):
## Model
anthropic/claude-sonnet-4-6OpenAI:
## Model
openai/gpt-4oOllama (local, no API key needed):
## Model
ollama/qwen3:32bMix providers per role — e.g. cheap local model for development, strong cloud model for evaluation:
## Model
- develop: ollama/qwen3:32b
- evaluate: anthropic/claude-sonnet-4-6
- improve: ollama/qwen3:32b| Provider | Model format | API key env var |
|---|---|---|
| Anthropic | anthropic/claude-sonnet-4-6 |
ANTHROPIC_API_KEY |
| OpenAI | openai/gpt-4o |
OPENAI_API_KEY |
| Ollama | ollama/qwen3:32b |
None (local) |
| DeepSeek | deepseek/deepseek-chat |
DEEPSEEK_API_KEY |
| Groq | groq/llama-3.3-70b-versatile |
GROQ_API_KEY |
| Together AI | together_ai/meta-llama/... |
TOGETHERAI_API_KEY |
| Azure OpenAI | azure/gpt-4o |
AZURE_API_KEY |
| Google Gemini | gemini/gemini-2.0-flash |
GEMINI_API_KEY |
See the full list at LiteLLM Providers.
- Install Ollama and pull a model:
ollama pull qwen3:32b
- Set the model in
program.md:## Model ollama/qwen3:32b - Run AutoDev — no API key needed.
Tool calling: Models with native function-calling support (Qwen, Llama 3.1+, Mistral Large) use it automatically. For models without it, AutoDev falls back to prompt-based tool calling — tool definitions are injected into the system prompt and JSON responses are parsed.
Vision: Visual evaluation requires a vision-capable model (Claude, GPT-4o, Gemini, LLaVA). If the configured model doesn't support vision, the visual score defaults to 0.5 with a note. You can set Visual: 0% in weights to disable it entirely, or use a per-role config to route evaluation to a vision model.
program.md # Human control surface — goals, weights, MCP configs
run.py # Orchestrator — the autonomous loop
develop.py # Builder agent — Claude API tool-use loop
evaluate.py # Evaluation harness — Playwright + Vision (immutable)
improve.py # Feedback agent — targeted repairs from eval results
server.py # MCP server — exposes AutoDev as tools for Claude Code
api.py # FastAPI backend for the React GUI
gui/ # React dashboard (Kanban board, score charts, live activity)
workspace/ # Where the software gets built (auto-created, gitignored)
results.tsv # Experiment log (auto-generated)
tasks.json # Decomposed tasks + state (auto-generated)
run.py— The orchestrator. Parsesprogram.md, manages git commits inworkspace/, connects MCP servers, and runs the main DEVELOP → EVALUATE → IMPROVE → RE-EVALUATE → DECIDE cycle. Emits WebSocket events for real-time GUI updates.develop.py— The builder agent. Runs a Claude API tool-use loop with filesystem tools (read, write, list, run commands) plus any MCP tools. Also handles goal decomposition — breaking high-level goals into discrete, testable tasks with priorities and dependencies.evaluate.py— The evaluation harness. Immutable by design (never modified during runs) to ensure objective scoring. Launches the workspace app with Playwright, runs interaction tests, captures screenshots for Claude Vision analysis, measures performance via the Performance API, and checks accessibility via ARIA snapshots.improve.py— The repair agent. Takes concrete evaluation evidence (console errors, failing interactions, visual defects) and makes targeted fixes. No new features — only repairs what the evaluator flagged.api.py— FastAPI backend serving REST endpoints for program management, task CRUD, loop control (start/stop/pause/resume), workspace browsing, and WebSocket relay for real-time events.server.py— FastMCP server that exposes AutoDev as MCP tools (start_autodev,get_status,stop_autodev,set_program,get_tasks, etc.) for integration with Claude Code or other MCP clients.
Built with React 18, TypeScript, Vite, and Tailwind CSS. Key dependencies include Recharts for score visualization, Framer Motion for animations, @dnd-kit for drag-and-drop kanban, @monaco-editor/react for the program editor, and lucide-react for icons.
The frontend connects to the API via a Vite proxy (/api → localhost:8000, /ws → WebSocket) and uses a custom useAutoDevSocket hook for real-time event streaming with auto-reconnect.
Add domain-specific capabilities by declaring MCP servers in program.md:
[
{"name": "playwright", "command": "npx", "args": ["@playwright/mcp"]},
{"name": "godot", "command": "python", "args": ["-m", "mcp_godot"]},
{"name": "blender", "command": "python", "args": ["-m", "mcp_blender"]},
{"name": "sqlite", "command": "npx", "args": ["@anthropic-ai/mcp-sqlite", "--db", "app.db"]}
]Tools are auto-discovered and namespaced (e.g., playwright__screenshot). Adding a new MCP server = one JSON line.
At startup, Claude decomposes your goal into discrete, testable tasks with dependencies. The orchestrator works through them in priority order:
- Tasks pass with composite score > 0.7
- Tasks are retried up to 3 times before being marked as blocked
- Humans can reorder, add, remove, or unblock tasks via the GUI
- Dependencies are respected — a task won't start until its prerequisites are done
MIT




