feat(pi): add Pi harness (--type pi) with docker + skill injection

.claude/harness-candidates.md

@@ -455,3 +455,12 @@ with the two `action.yml` items above — one considered change to the action's
   `final_status`, which does not exist in `run.json` and would have made a new
   assertion dead on arrival. Guard: assert the key set that non-Python consumers
   depend on, mirroring how CE030 pins doc/schema parity.
+
+- [ ] A `_*TurnState` (agent turn-state) attribute that is written but never read
+  outside its own assignment — CE037-class dead accumulator. Surfaced during the
+  Pi harness port: `_PiTurnState.turns_finished` was copied from OpenCode's
+  `steps_finished` (which drives that agent's `finished_without_tokens` guard) but
+  Pi deliberately dropped that guard, leaving the counter dead. Fixed by hand this
+  run. Guard would need cross-method dataflow over each `Agent`-subclass turn-state
+  class (write sites vs read sites), which the AST-only CExxx runner can't express
+  in ~30 min — deferred. Caught in: Pi harness Phase 2 quality review.

.github/pages-stub/index.html

@@ -12,7 +12,7 @@
     -->
     <meta
       name="description"
-      content="Coder Eval is Playwright for coding agents: an open-source, agent-agnostic framework for evaluating and benchmarking AI coding agents and their skills — sandboxed runs of Claude Code, Codex, Antigravity (Gemini), or OpenCode against declarative YAML tasks, with weighted scoring and CI gates. Documentation: coder-eval.com/docs"
+      content="Coder Eval is Playwright for coding agents: an open-source, agent-agnostic framework for evaluating and benchmarking AI coding agents and their skills — sandboxed runs of Claude Code, Codex, Antigravity (Gemini), OpenCode, or Pi against declarative YAML tasks, with weighted scoring and CI gates. Documentation: coder-eval.com/docs"
     />
 
     <!--
@@ -227,7 +227,7 @@ <h1 class="sr-only">Coder Eval</h1>
       <p class="lead">
         <strong>Playwright for coding agents.</strong> An open-source, agent-agnostic framework for
         evaluating and benchmarking AI coding agents and their skills: it runs a real agent — Claude
-        Code, Codex, Antigravity (Gemini), or OpenCode — in a sandbox against declarative YAML
+        Code, Codex, Antigravity (Gemini), OpenCode, or Pi — in a sandbox against declarative YAML
         tasks, then scores the files and commands the agent actually produced.
       </p>
       <p class="notice">

README.md

@@ -17,14 +17,14 @@
 open-source, **agent-agnostic** framework for **evaluating and benchmarking AI coding
 agents and their skills** — built for benchmark authors, CLI builders, and skill
 builders — with sandboxing, reproducibility, and data-driven analysis. It runs a real
-agent — **Claude Code**, **OpenAI Codex**, **Google Antigravity (Gemini)**, or
-**OpenCode** — in a sandbox against declarative YAML tasks, then scores the files and
+agent — **Claude Code**, **OpenAI Codex**, **Google Antigravity (Gemini)**,
+**OpenCode**, or **Pi** — in a sandbox against declarative YAML tasks, then scores the files and
 commands it actually produced. Changing harness is one field (`agent.type`); the
 tasks, criteria, scoring, telemetry, and reports stay the same.
 
 Reach for it when you want to **benchmark agents on your own domain tasks**,
 **test whether a skill triggers** in the agent you ship for, **A/B-test Claude Code
-vs. Codex vs. Gemini vs. OpenCode** (or model vs. model, prompt vs. prompt), or
+vs. Codex vs. Gemini vs. OpenCode vs. Pi** (or model vs. model, prompt vs. prompt), or
 **gate CI on coding-agent quality**. It is **not a fixed leaderboard**: unlike
 SWE-bench or SkillsBench, which rank models on a shared task set, you bring the tasks
 and you bring the scoring — weighted 0.0–1.0 criteria, a `skill_triggered` activation
@@ -40,7 +40,7 @@ care about. See [How it compares](https://coder-eval.com/docs/comparison).
 - **Sandboxed execution** in isolated environments with resource limits
 - **Weighted, continuous scoring** (0.0–1.0) with fractional credit and thresholds
 - **Many criterion types** — from file checks to code similarity and LLM-graded rubrics
-- **Agent-agnostic by design** — Claude Code, OpenAI Codex, Antigravity (Gemini), and OpenCode today; add your own harness through the plugin SPI
+- **Agent-agnostic by design** — Claude Code, OpenAI Codex, Antigravity (Gemini), OpenCode, and Pi today; add your own harness through the plugin SPI
 - **Experiment layer** — A/B agent configs (models, tools, prompts) side-by-side
 - **Full telemetry** — every tool call, token counts, and cost, with real-time streaming
 
@@ -62,7 +62,7 @@ care about. See [How it compares](https://coder-eval.com/docs/comparison).
 ## What you can do with it
 
 - **Benchmark coding agents** — score an agent across a suite of tasks with weighted scoring and pass/fail thresholds
-- **Compare models & configs** — A/B-test Claude vs. Codex vs. Gemini vs. OpenCode, model vs. model, tool-on vs. tool-off, prompt vs. prompt
+- **Compare models & configs** — A/B-test Claude vs. Codex vs. Gemini vs. OpenCode vs. Pi, model vs. model, tool-on vs. tool-off, prompt vs. prompt
 - **Evaluate skills** — verify an agent actually engages a target skill (`skill_triggered`) and score skill-driven suites (SkillsBench-style), on whichever harness your users run
 - **Keep skills up to date in CI** — re-validate your skills on every change or on a schedule; catch silent regressions when models, prompts, or the skills themselves drift
 - **Gate CI on agent quality** — run the suite in GitHub Actions and fail the build on regressions
@@ -252,6 +252,7 @@ The step's exit code is coder-eval's own: non-zero on any failed task.
 | [Codex](docs/agents/CODEX.md) | Running the OpenAI Codex agent |
 | [Antigravity (Gemini)](docs/agents/ANTIGRAVITY.md) | Running the Google Antigravity / Gemini agent |
 | [OpenCode](docs/agents/OPENCODE.md) | Running the OpenCode agent on open-weight models |
+| [Pi](docs/agents/PI.md) | Running the Pi agent on open-weight models |
 | [Run-Limit Parity](docs/agents/HARNESS_PARITY.md) | What each run_limits field means on every harness |
 | [A/B Experiments](docs/AB_EXPERIMENTS.md) | Compare models / tools / prompts across the same tasks |
 | [Bring Your Own Dataset](docs/DATASETS.md) | Fan a single task out over a dataset |

docker/Dockerfile

@@ -36,6 +36,12 @@ RUN curl -fsSL https://deb.nodesource.com/setup_22.x | bash - \
     && rm -rf /var/lib/apt/lists/* \
     && npm install -g @anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}
 
+# Pi Node CLI, pinned — same rationale as the Claude Code pin above (the agent
+# binary is a dominant non-model driver of results; @latest would freeze
+# nondeterministically under layer caching). Node 22 + npm are already present.
+ARG PI_VERSION=0.84.4
+RUN npm install -g @earendil-works/pi-coding-agent@${PI_VERSION}
+
 # uv: matches host sandbox.py's `uv venv` + `uv pip install` fast path
 RUN curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR=/usr/local/bin sh
 
@@ -58,12 +64,16 @@ COPY experiments/default.yaml ./experiments/default.yaml
 # come from public PyPI, so this needs no private-index credentials. The RUN
 # below always passes `--extra codex --extra antigravity --extra litellm`.
 #
-# NOT every built-in agent ships here: `opencode` is registered unconditionally
-# but its CLI is a Node package (`npm install -g opencode-ai`), absent from this
-# image, and no OpenCode credentials are in SandboxConfig.env_passthrough -- so
-# `--driver docker` does not support it. Adding it means a pinned version that
-# travels with the release tag (as CLAUDE_CODE_VERSION does) plus an
-# env_passthrough block; see docs/agents/OPENCODE.md "Running in Docker".
+# `pi` IS baked above (pinned PI_VERSION) and its OPENROUTER_API_KEY provider
+# credential is in DockerDriverConfig.env_passthrough, so `--driver docker --type pi`
+# is supported. NOT every built-in agent ships here, though: `opencode` is
+# registered unconditionally but its CLI is a Node package
+# (`npm install -g opencode-ai`), absent from this image -- so `--driver docker`
+# does not support it. (OPENROUTER_API_KEY IS now in env_passthrough, added for Pi,
+# so an OpenCode env-auth run would authenticate once the CLI is baked; the CLI is
+# the remaining blocker.) Adding it means a pinned version that travels with the release
+# tag (as CLAUDE_CODE_VERSION / PI_VERSION do) plus an env_passthrough block; see
+# docs/agents/OPENCODE.md "Running in Docker".
 #
 # CODER_EVAL_UV_EXTRAS carries ADDITIONAL opt-in extras on top of those; it
 # defaults to none. `make docker-image-full` passes `--extra uipath`, which

docs/DOCKER_ISOLATION.md

@@ -15,7 +15,7 @@ Set `sandbox.driver: docker` on a task (or pass `--driver docker` on the CLI —
 a thin alias for `-D sandbox.driver=docker`) when you want:
 
 - **Isolation from the host filesystem/network** — agent-generated code can't reach files outside the sandbox.
-- **A pinned toolchain** — the image bakes in Python 3.13, Node 22 LTS, `@anthropic-ai/claude-code`, `uv`, and the matching `coder_eval` version, so results don't drift with host upgrades.
+- **A pinned toolchain** — the image bakes in Python 3.13, Node 22 LTS, `@anthropic-ai/claude-code`, the `pi` CLI (`@earendil-works/pi-coding-agent`), the codex/antigravity/litellm agent SDKs, `uv`, and the matching `coder_eval` version, so results don't drift with host upgrades.
 
 Aggregation (P/R/F1, suite thresholds, reports) always stays on the host. Each container is a sealed "run one task → emit one `task.json`" worker.
 

docs/EXTENDING.md

@@ -285,8 +285,8 @@ The base package ships **no** plugin rates; only the built-in table.
 ## See also
 
 - [Claude Code](agents/CLAUDE_CODE.md) · [Codex](agents/CODEX.md) ·
-  [Antigravity](agents/ANTIGRAVITY.md) · [OpenCode](agents/OPENCODE.md) — the
-  built-in agents, each registered via this same SPI
+  [Antigravity](agents/ANTIGRAVITY.md) · [OpenCode](agents/OPENCODE.md) ·
+  [Pi](agents/PI.md) — the built-in agents, each registered via this same SPI
 - [Task Definition Guide](TASK_DEFINITION_GUIDE.md) — the criterion catalogue
 - [CLAUDE.md](https://github.com/UiPath/coder_eval/blob/main/CLAUDE.md) — architecture
   and extension points in depth

docs/USER_GUIDE.md

@@ -39,7 +39,7 @@ coder-eval run tasks/hello_date.yaml --stream full  # live LLM output
 | `-D path=value` / `--set` | Override any resolved task-config field (`agent`/`run_limits`/`sandbox` roots), e.g. `-D run_limits.max_turns=30 -D agent.permission_mode=plan -D agent.sdk_options.effort=high`. Repeatable; schema-validated. This is the way to set permission mode, turn/timeout limits, token/USD budget caps, tools, plugins, and SDK options. |
 | `--model, -m` | Shorthand alias for `-D agent.model=…` (e.g., `claude-sonnet-5`) |
 | `--driver` | Shorthand alias for `-D sandbox.driver=…` (`tempdir` or `docker`) |
-| `--type, -T` | Override agent type for all tasks (`claude-code`, `codex`, `antigravity`, `opencode`, or a plugin kind). |
+| `--type, -T` | Override agent type for all tasks (`claude-code`, `codex`, `antigravity`, `opencode`, `pi`, or a plugin kind). |
 | `--repeats` | Run each `(task, variant)` N times (≥1); overrides experiment/variant `repeats:`. See [Replicates](#replicates). |
 | `--resume` | Resume an interrupted run: skip tasks already finalized in `--run-dir` and run the rest, folding prior results into `run.json`. Requires `--run-dir`. A task with *any* final status (incl. FAILED/ERROR) counts as finalized, so resume does **not** retry failures — delete a task's `task.json` to force a re-run. A config mismatch is warned, not refused. |
 | `--sample N` | For dataset-backed tasks, run a fixed-seed random N-row sample (reproducible; cheap smoke test). See [Bring Your Own Dataset](DATASETS.md). |

docs/agents/HARNESS_PARITY.md

@@ -10,12 +10,12 @@ This page is the contract for what each run limit means per harness, plus the sh
 
 ## The table
 
-| Limit | claude-code | codex | antigravity | opencode |
-|---|---|---|---|---|
-| `run_limits.max_turns` | native SDK cap (agent-loop turns) | visible-turn cap (resolved tool calls) | visible-turn cap (resolved tool calls) | native step cap (the CLI's own agent-loop steps) |
-| `run_limits.turn_timeout` | watchdog, SIGKILL on the CLI subprocess | watchdog + cooperative interrupt | watchdog, plus an earlier internal poll deadline at 80% of it (see below) | deadline enforced in-loop and on the final reap; SIGTERM→SIGKILL on the CLI's whole process group |
-| `run_limits.task_timeout` | orchestrator-level, agent-agnostic | orchestrator-level, agent-agnostic | orchestrator-level, agent-agnostic | orchestrator-level, agent-agnostic |
-| `run_limits.stop_early` | cooperative `should_stop` | cooperative `should_stop` | cooperative `should_stop` | cooperative `should_stop` (event granularity) |
+| Limit | claude-code | codex | antigravity | opencode | pi |
+|---|---|---|---|---|---|
+| `run_limits.max_turns` | native SDK cap (agent-loop turns) | visible-turn cap (resolved tool calls) | visible-turn cap (resolved tool calls) | native step cap (the CLI's own agent-loop steps) | native turn cap (the CLI's own `turn_start` agent-loop steps) |
+| `run_limits.turn_timeout` | watchdog, SIGKILL on the CLI subprocess | watchdog + cooperative interrupt | watchdog, plus an earlier internal poll deadline at 80% of it (see below) | deadline enforced in-loop and on the final reap; SIGTERM→SIGKILL on the CLI's whole process group | deadline enforced in-loop and on the final reap; SIGTERM→SIGKILL on the CLI's whole process group |
+| `run_limits.task_timeout` | orchestrator-level, agent-agnostic | orchestrator-level, agent-agnostic | orchestrator-level, agent-agnostic | orchestrator-level, agent-agnostic | orchestrator-level, agent-agnostic |
+| `run_limits.stop_early` | cooperative `should_stop` | cooperative `should_stop` | cooperative `should_stop` | cooperative `should_stop` (event granularity) | cooperative `should_stop` (event granularity — Pi streams incrementally) |
 
 ## `max_turns` counts visible turns on Codex and Antigravity
 
@@ -50,6 +50,14 @@ when step N+1 begins, with the completed steps' tokens intact. A step is one
 assistant generation and may carry several tool calls — so, as with claude-code,
 the same number is a looser tool-call budget than on the visible-turn backends.
 
+**Pi keeps a native unit too — its `turn_start` agent-loop steps.** Like OpenCode,
+`pi -p --mode json` runs a real multi-step agent loop per invocation and streams it
+(`turn_start` / `turn_end`), so `max_turns: N` allows N complete turns and cuts the
+run when turn N+1 begins, with the completed turns' tokens intact. Pi streams
+incrementally, so the cut genuinely stops spend mid-run. A Pi turn is one assistant
+generation and may carry several tool calls — the same looser budget as claude-code
+and OpenCode.
+
 **So holding `max_turns` constant across harnesses does not hold the budget
 constant.** If you are A/B-ing across backends and the cap is close to binding, that
 is the number to distrust.
@@ -116,17 +124,26 @@ whose cap fires should not look like a task whose harness hung.
 Not a run limit, but the same promise: one task file, three harnesses, same meaning.
 This field breaks it silently.
 
-| | claude-code | codex | antigravity |
-|---|---|---|---|
-| `<path>/skills/<name>/SKILL.md` (plugin root) | **required** | accepted | accepted |
-| `<path>/<name>/SKILL.md` (bare skills dir) | **loads nothing** | accepted | accepted |
+| | claude-code | codex | antigravity | pi |
+|---|---|---|---|---|
+| `<path>/skills/<name>/SKILL.md` (plugin root) | **required** | accepted | accepted | accepted |
+| `<path>/<name>/SKILL.md` (bare skills dir) | **loads nothing** | accepted | accepted | **loads, but undetected** † |
 
 claude-code hands the value to the SDK as a *plugin directory*, and a plugin's skills
 live at `<plugin>/skills/<name>/SKILL.md`. Point it at the directory that directly
 parents the skill directories and no skill loads. Codex
 (`codex_agent._setup_skills`) and Antigravity (`antigravity_agent._resolve_skills_paths`)
 both scan **both** layouts and take whichever actually holds a `<skill>/SKILL.md`.
 
+† Pi uses the shared `_plugin_skill_dirs` resolver, whose bare-dir fallback resolves a
+bare skills directory to itself and passes it as `--skill <dir>`, so the skill *does*
+load and the agent can use it. But `skill_triggered` detects engagement by matching a
+`skills/<name>/` segment in the read path (`_SKILL_PATH_RE`), which a bare dir lacks — so
+an **activation suite** on a bare dir still scores recall 0 even though the skill ran.
+Net effect for activation suites is therefore the same silent-0 as claude-code, via a
+different mechanism; use the plugin-root shape (lint rule CE045 holds `SKILL_SOURCE_PATH`
+to it for exactly this reason).
+
 So `.claude/skills` works on two backends out of three and fails on the third — and
 fails without an error. The agent simply is not offered the skill, every positive row
 of an activation suite scores 0, and the suite reports recall 0.0. That is
@@ -165,15 +182,45 @@ plugin-root shape by lint rule CE045. The rule keys on that variable name only;
 one, feeds `experiments/plugin-comparison.yaml`, whose default agent is claude-code,
 so the same requirement applies there and is unlinted.
 
+**OpenCode and Pi both honor the *skills* half of a plugin.** OpenCode maps each
+local plugin root to its `skills.paths`; Pi maps each to a `--skill <dir>` argument —
+both via the same `_plugin_skill_dirs` resolver — so both **can** run activation
+suites. A plugin's non-skill assets (agents/hooks/commands/MCP servers) are dropped on
+both. See [OpenCode](OPENCODE.md) and [Pi § plugins](PI.md#known-limitations).
+
+## Pi enforces `system_prompt` but not the tool allowlists
+
+- **`system_prompt` is ENFORCED** (`--append-system-prompt`, semantics `append`) — a
+  small win over OpenCode, which drops it.
+- **`allowed_tools` / `disallowed_tools` are NOT enforced.** Pi's built-in tools are
+  lowercase (`bash`/`read`/`write`/`edit`/`grep`/`find`/`ls`), but the shared config
+  default (`experiments/default.yaml`) sets Claude-namespaced names
+  (`Bash`/`Read`/`Write`/…). Forwarding those to `--tools` would allowlist tools that
+  do not exist in Pi and strip the agent of ALL tools — so, like OpenCode (drops them),
+  Codex (forwards `disallowed_tools` without SDK enforcement), and Antigravity (does not
+  read them), Pi ignores them and runs with its full native toolset. A task that needs a
+  restricted Pi toolset would have to name Pi's lowercase tools — a documented follow-up.
+- **`permission_mode` is NOT enforced** — Pi headless print mode auto-runs tools and
+  exposes only project-file trust (`--approve` / `--no-approve`), no tool-approval
+  mode; the sandbox driver is the isolation boundary (same as Codex/Antigravity).
+- **`system_prompt_file` is NOT read** (use inline `system_prompt`), matching
+  Codex/Antigravity.
+- **Built-in auto-retry.** Pi retries a transient/provider error *internally* (another
+  `agent_start` cycle in the same invocation, flagged `willRetry: true`), which the
+  harness folds into one turn. The internal retry is bounded by
+  `turn_timeout` / `task_timeout`.
+
+Full detail: [Pi](PI.md).
+
 ## Reproducing
 
 `tasks/run_limits/` holds one fixture per limit: `max_turns_cap.yaml` asks for more
 sequential work than its cap allows, and `turn_timeout.yaml` runs a command that
 outlives its watchdog. Run either with `--type claude-code` / `--type codex` /
-`--type antigravity` / `--type opencode` to check a backend against the contract
-above.
+`--type antigravity` / `--type opencode` / `--type pi` to check a backend against the
+contract above.
 
 ## Related
 
-- [Claude Code](CLAUDE_CODE.md) · [Codex](CODEX.md) · [Antigravity](ANTIGRAVITY.md) · [OpenCode](OPENCODE.md)
+- [Claude Code](CLAUDE_CODE.md) · [Codex](CODEX.md) · [Antigravity](ANTIGRAVITY.md) · [OpenCode](OPENCODE.md) · [Pi](PI.md)
 - [Task Definition Guide](../TASK_DEFINITION_GUIDE.md) — the full `run_limits` schema