Skip to content
cheetohsumPublic

About

Autonomous Software Development via Iterative Evaluation

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

AutoDev

Autonomous software development via iterative evaluation. Inspired by Karpathy's autoresearch — an AI agent that autonomously builds, evaluates, and iterates on software applications using objective quality metrics.

Instead of training a model overnight, AutoDev builds software overnight: write code → start server → evaluate with Playwright + Claude Vision → fix issues → keep or discard → repeat.

How It Works

┌─────────────────────────────────────────────────────┐
│                   program.md                         │
│         (goal, requirements, weights, MCP)           │
└──────────────────────┬──────────────────────────────┘
                       │
                       ▼
              ┌─── DECOMPOSE ───┐
              │  Goal → Tasks   │
              │  (Kanban board) │
              └────────┬────────┘
                       │
          ┌────────────▼────────────┐
          │     SELECT NEXT TASK    │◄──────────────┐
          └────────────┬────────────┘               │
                       │                            │
          ┌────────────▼────────────┐               │
          │        DEVELOP          │               │
          │  (Claude + tools)       │               │
          └────────────┬────────────┘               │
                       │                            │
          ┌────────────▼────────────┐               │
          │       EVALUATE          │               │
          │  Playwright + Vision    │               │
          └────────────┬────────────┘               │
                       │                            │
          ┌────────────▼────────────┐               │
          │        IMPROVE          │               │
          │  Fix specific issues    │               │
          └────────────┬────────────┘               │
                       │                            │
          ┌────────────▼────────────┐               │
          │      RE-EVALUATE        │               │
          └────────────┬────────────┘               │
                       │                            │
          ┌────────────▼────────────┐    score >    │
          │        DECIDE           │────best?──────┘
          │  keep / revert / block  │   repeat
          └─────────────────────────┘

Each iteration gets two shots — an initial build and a repair pass driven by concrete Playwright evidence (console errors, failing interactions, visual defects, accessibility violations).

The Dashboard

AutoDev ships with a real-time React dashboard for monitoring and controlling the autonomous loop. The GUI connects to the FastAPI backend over REST + WebSocket, giving you live visibility into every iteration.

Setup Screen

On first launch, the setup screen guides you through configuring program.md — the single human control surface that tells AutoDev what to build. A Monaco-powered markdown editor lets you define your goal, requirements, evaluation weights, and MCP server configurations. Click Save & Decompose Tasks and Claude breaks your goal into a prioritized task plan.

Setup Screen

Kanban Board

The primary view for managing the development pipeline. Tasks flow through six columns — Backlog, In Progress, Evaluating, Improving, Done, and Blocked — as the autonomous loop works through them. You can drag-and-drop to reprioritize, add custom tasks, filter by name, and manually unblock stuck tasks. The progress bar at the top shows overall completion. Color-coded priority dots (red = high, orange = medium, blue = low) give quick visual triage. The right panel shows live agent activity and evaluation screenshots as they happen.

Kanban Board

Metrics & Iteration Log

Track quality progression across every iteration. The Score History chart plots composite and per-dimension scores over time, with toggles for each evaluation dimension (Functionality, Visual, Performance, Accessibility, Structure, Errors). Green markers indicate kept iterations; red markers show reverted ones. Below the chart, the Iteration Log table provides a detailed breakdown of every iteration — which task was attempted, individual dimension scores, score deltas, and whether the change was kept or discarded.

Metrics Tab

Workspace Browser

A split-pane file explorer and source viewer for the software being built. The left panel shows the workspace file tree (auto-refreshed), and clicking any file renders its contents in the right panel. Useful for inspecting what the agent has written without leaving the dashboard.

Workspace Tab

Settings & Program Editor

Edit program.md live without restarting the loop. Changes take effect on the next iteration — the orchestrator re-reads the program file at the start of each cycle. The Monaco editor provides syntax highlighting, line numbers, and word wrap for comfortable editing. Below the editor, the MCP Servers panel shows which tool servers are configured and connected, with tool counts per server.

Settings Tab

Evaluation Dimensions

Dimension What it measures
Functionality Playwright interaction testing, console errors
Visual Claude Vision screenshot analysis against quality criteria
Performance window.performance API — load times, DOM size
Accessibility ARIA snapshot, heading structure, alt text, labels
Structure File organization, no giant files, separation of concerns
Errors Console error count penalty

Weights are configurable in program.md and can be adjusted live while the loop runs.

Quick Start

Prerequisites

Install

pip install -e .
playwright install chromium
cd gui && npm install && cd ..

Run

  1. Edit program.md — describe what you want to build

  2. Start the loop:

    python run.py
  3. Or with the GUI:

    # Terminal 1: API server
    python api.py
    # Terminal 2: React dev server
    cd gui && npm run dev

    Open http://localhost:5174

  4. Or as an MCP server (for Claude Code):

    python server.py

Environment Variables

Variable Default Description
ANTHROPIC_API_KEY — Anthropic API key (if using Claude)
OPENAI_API_KEY — OpenAI API key (if using GPT models)
AUTODEV_MODEL anthropic/claude-sonnet-4-6 Global model override (LiteLLM format)
AUTODEV_MAX_ITERATIONS 100 Maximum iterations

Multi-Provider Support

AutoDev supports any LLM provider via LiteLLM. Configure the model in program.md using provider/model-name format:

Quick Examples

Anthropic (default):

## Model
anthropic/claude-sonnet-4-6

OpenAI:

## Model
openai/gpt-4o

Ollama (local, no API key needed):

## Model
ollama/qwen3:32b

Mix providers per role — e.g. cheap local model for development, strong cloud model for evaluation:

## Model
- develop: ollama/qwen3:32b
- evaluate: anthropic/claude-sonnet-4-6
- improve: ollama/qwen3:32b

Supported Providers

Provider Model format API key env var
Anthropic anthropic/claude-sonnet-4-6 ANTHROPIC_API_KEY
OpenAI openai/gpt-4o OPENAI_API_KEY
Ollama ollama/qwen3:32b None (local)
DeepSeek deepseek/deepseek-chat DEEPSEEK_API_KEY
Groq groq/llama-3.3-70b-versatile GROQ_API_KEY
Together AI together_ai/meta-llama/... TOGETHERAI_API_KEY
Azure OpenAI azure/gpt-4o AZURE_API_KEY
Google Gemini gemini/gemini-2.0-flash GEMINI_API_KEY

See the full list at LiteLLM Providers.

Local Models with Ollama

  1. Install Ollama and pull a model:
    ollama pull qwen3:32b
  2. Set the model in program.md:
    ## Model
    ollama/qwen3:32b
  3. Run AutoDev — no API key needed.

Tool calling: Models with native function-calling support (Qwen, Llama 3.1+, Mistral Large) use it automatically. For models without it, AutoDev falls back to prompt-based tool calling — tool definitions are injected into the system prompt and JSON responses are parsed.

Vision: Visual evaluation requires a vision-capable model (Claude, GPT-4o, Gemini, LLaVA). If the configured model doesn't support vision, the visual score defaults to 0.5 with a note. You can set Visual: 0% in weights to disable it entirely, or use a per-role config to route evaluation to a vision model.

Project Structure

program.md        # Human control surface — goals, weights, MCP configs
run.py            # Orchestrator — the autonomous loop
develop.py        # Builder agent — Claude API tool-use loop
evaluate.py       # Evaluation harness — Playwright + Vision (immutable)
improve.py        # Feedback agent — targeted repairs from eval results
server.py         # MCP server — exposes AutoDev as tools for Claude Code
api.py            # FastAPI backend for the React GUI
gui/              # React dashboard (Kanban board, score charts, live activity)
workspace/        # Where the software gets built (auto-created, gitignored)
results.tsv       # Experiment log (auto-generated)
tasks.json        # Decomposed tasks + state (auto-generated)

Architecture

Python Backend

  • run.py — The orchestrator. Parses program.md, manages git commits in workspace/, connects MCP servers, and runs the main DEVELOP → EVALUATE → IMPROVE → RE-EVALUATE → DECIDE cycle. Emits WebSocket events for real-time GUI updates.
  • develop.py — The builder agent. Runs a Claude API tool-use loop with filesystem tools (read, write, list, run commands) plus any MCP tools. Also handles goal decomposition — breaking high-level goals into discrete, testable tasks with priorities and dependencies.
  • evaluate.py — The evaluation harness. Immutable by design (never modified during runs) to ensure objective scoring. Launches the workspace app with Playwright, runs interaction tests, captures screenshots for Claude Vision analysis, measures performance via the Performance API, and checks accessibility via ARIA snapshots.
  • improve.py — The repair agent. Takes concrete evaluation evidence (console errors, failing interactions, visual defects) and makes targeted fixes. No new features — only repairs what the evaluator flagged.
  • api.py — FastAPI backend serving REST endpoints for program management, task CRUD, loop control (start/stop/pause/resume), workspace browsing, and WebSocket relay for real-time events.
  • server.py — FastMCP server that exposes AutoDev as MCP tools (start_autodev, get_status, stop_autodev, set_program, get_tasks, etc.) for integration with Claude Code or other MCP clients.

React Frontend (gui/)

Built with React 18, TypeScript, Vite, and Tailwind CSS. Key dependencies include Recharts for score visualization, Framer Motion for animations, @dnd-kit for drag-and-drop kanban, @monaco-editor/react for the program editor, and lucide-react for icons.

The frontend connects to the API via a Vite proxy (/api → localhost:8000, /ws → WebSocket) and uses a custom useAutoDevSocket hook for real-time event streaming with auto-reconnect.

Modular MCP Architecture

Add domain-specific capabilities by declaring MCP servers in program.md:

[
  {"name": "playwright", "command": "npx", "args": ["@playwright/mcp"]},
  {"name": "godot", "command": "python", "args": ["-m", "mcp_godot"]},
  {"name": "blender", "command": "python", "args": ["-m", "mcp_blender"]},
  {"name": "sqlite", "command": "npx", "args": ["@anthropic-ai/mcp-sqlite", "--db", "app.db"]}
]

Tools are auto-discovered and namespaced (e.g., playwright__screenshot). Adding a new MCP server = one JSON line.

Task System

At startup, Claude decomposes your goal into discrete, testable tasks with dependencies. The orchestrator works through them in priority order:

  • Tasks pass with composite score > 0.7
  • Tasks are retried up to 3 times before being marked as blocked
  • Humans can reorder, add, remove, or unblock tasks via the GUI
  • Dependencies are respected — a task won't start until its prerequisites are done

License

MIT

About

Autonomous Software Development via Iterative Evaluation

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages