coding-agent-bench-plus

Built on affaan-m/ECC by @affaan-m (239,455 stars, MIT). All credit for the original idea to them. This fork improves and repackages it; upstream license preserved in UPSTREAM_LICENSE.

A Claude Code skill to compare coding agents on the same real tasks, using pass rate, cost, time, and repeat results.

🎯 Why

For teams choosing between Claude Code, Aider, and Codex.

Run each agent from the same Git commit.

Use the same task and the same checks every time.

Get facts before you pick a tool or model.

⚡ Install

Read this repo first. Then install the skill:

mkdir -p ~/.claude/skills/agent-eval && curl -fsSL https://raw.githubusercontent.com/your-org/coding-agent-bench-plus/main/SKILL.md -o ~/.claude/skills/agent-eval/SKILL.md

Use a clean Git repo with a known commit. Do not test on private data, real payments, or live production systems.

🧪 Usage

Ask Claude Code to create a task file, then run the same task with each agent:

mkdir -p tasks
agent-eval run \
  --task tasks/add-retry-logic.yaml \
  --agent claude-code \
  --agent aider \
  --runs 3

agent-eval report --format table

Expected output looks like this:

Task: add-retry-logic (3 runs each)

Agent        Pass Rate   Cost    Time   Consistency
claude-code  3/3         $0.12   45s    100%
aider        2/3         $0.08   38s     67%

A task file should lock one repo commit and include clear checks. Use tests or builds when you can. A grep check only shows that text exists. It does not prove the code works.

Each run uses its own Git worktree. If setup fails, check that the repo is Git, the commit exists, the target folder is writable, and no old failed worktree remains.

Do not count missing cost data as zero. Mark it as unavailable.

🔁 What we changed vs upstream

• 英語化され、説明がより平易で実務向けの表現になった。
• インストール前の確認、クリーンなGitリポジトリ、私有データ・本番環境を避ける安全上の注意が追加された。
• worktree作成の失敗原因と、コスト未取得をゼロ扱いしない注意が追加された。
• 判定コマンドの反復実行安全性や、grep判定の限界を明確にした。
• 改善版は末尾が「chec」で途切れており、モデル判定以降の内容が欠落している。

📄 License

MIT. The upstream license is preserved in UPSTREAM_LICENSE.