diff --git a/docs/COMMANDS.md b/docs/COMMANDS.md index cba56d7..4997ca4 100644 --- a/docs/COMMANDS.md +++ b/docs/COMMANDS.md @@ -158,6 +158,14 @@ zo gates set MODE --project NAME `MODE` is one of: `supervised` (human approves every gate), `auto` (orchestrator approves unless ambiguous), `full-auto` (all gates auto-approved). +### zo status (control plane) + +When a project has a `plan-ledger.json` (v2 WS-B — generated automatically at +plan decompose), `zo status` renders progress from it first: per-phase status, +oracle-owned pass counts, attempts, and last failure. The ledger is the +machine-readable truth; STATE.md remains the human-readable projection and +the fallback for legacy projects. + ### zo gates approve / reject Record a nonce-verified human decision on the pending blocking gate (v2 diff --git a/memory/zo-platform/DECISION_LOG.md b/memory/zo-platform/DECISION_LOG.md index a47013b..84a4964 100644 --- a/memory/zo-platform/DECISION_LOG.md +++ b/memory/zo-platform/DECISION_LOG.md @@ -1265,3 +1265,15 @@ The `--no-headlines` flag is preserved (not removed) for backwards compatibility **Honest caveats:** PostToolUseFailure did not fire for nonzero-exit Bash commands in this session — its semantics appear limited to tool-infrastructure errors, so the failure feed will capture fewer events than designed; if nonzero-exit capture matters, add PostToolUse-with-error-inspection in WS-D. PreCompact/SessionEnd not yet observed live (no compaction occurred; SessionEnd fires at session close) — same wiring pattern as the three proven events. A full `zo build` demo run remains desirable on a machine with the claude CLI (this Desktop-managed Mac has none — see PR-046). **Outcome:** 904 → 908 tests (3 trace + 1 live-payload drift-guard), ruff clean. Fail-open verified end to end in production conditions. Evidence committed to PR #107. + +## Decision: 2026-08-12T17:00:00Z +**Type:** FEATURE + ARCHITECTURE +**Title:** v2 Phase 2 (WS-B control plane) — plan ledger with oracle-owned pass flags, story sizing lint, ledger-first status + +**Decision:** Implemented WS-B per plans/zo-v2-rearchitecture.md Phase 2. (1) `src/zo/ledger.py`: `plan-ledger.json` in the project memory root — one entry per (phase, subtask) with synthesized acceptance criteria (artifact existence for all phases; oracle threshold for phase_4), verification descriptor, `passes`, attempts, last_failure, plus a phase_status map. Regeneration is merge-preserving (progress survives re-decompose); every write atomic. (2) Oracle-owned flips: `mark_phase_passed` is called ONLY from the two verified-completion paths (automated gate after artifact/experiment checks, nonce-verified human PROCEED); ITERATE/loop-CONTINUE reset entries with the reason; `mark_subtask_complete` counts attempts but cannot set passes; the ledger file joined `_SEALED_DEFAULTS`, so builder Write/Edit is denied by the existing WS-A4 hook. (3) Sizing lint: optional `## Stories` plan section (StorySpec parser) + `validate_plan` rule — every declared story needs ≥1 machine-verifiable criterion (threshold/path/`command`); surfaces through preflight and `zo validate` for free; legacy plans without Stories are untouched. (4) `zo status` renders a control-plane table from the ledger (per-phase status, pass counts, attempts, last failure) with STATE.md as fallback/projection. + +**Also fixed (recon findings on Phase 1):** `contracts.set_active_phase` was non-atomic (torn read = enforcement silently off) — now temp+rename; the wrapper never exported hook env for delivery sessions — `zo build` now passes ZO_MEMORY_ROOT/ZO_DELIVERY_ROOT/ZO_CONTRACTS_PATH via extra_env, closing the per-project sealing gap. + +**Scope deferrals (deliberate, to Phase 3):** `evaluate_loop_state` keeps its ExperimentRegistry input (already oracle-derived; signature change touches ~20 test call sites and belongs to the fresh-context loop rework); session-state restore still reads STATE.md (cutover rides the fresh-context substrate to avoid regressing the PR-036 GATED-precedence resume fix). + +**Outcome:** 908 → 929 passed / 7 skipped (+21: 13 ledger, 5 stories/lint, 2 nonce-flip, 1 status), ruff clean, validate-docs green. Oracle checks 8-10 have passing seeded tests: status renders from the ledger; builder ledger-write denied while the oracle flip lands; vague story rejected. Cascade: specs/plan.md (§10 Stories + lint), specs/workflow.md (state-tracking line), docs/COMMANDS.md (status control plane). Branch `claude/v2-phase2-control-plane`. diff --git a/memory/zo-platform/STATE.md b/memory/zo-platform/STATE.md index 7f25ced..6ad190a 100644 --- a/memory/zo-platform/STATE.md +++ b/memory/zo-platform/STATE.md @@ -8,7 +8,7 @@ status: complete ## Current Position -**Session 040 (current) — pick up here.** Research + decision session: deep-dive review of three agent-orchestration repos (oh-my-claudecode, ruflo, ralph — cloned to `~/Documents/code/`) to inform the ZO v2 rearchitecture. 9-agent workflow (7 source-reading lenses + ZO baseline + adversarial synthesis, ~1.06M tokens) catalogued **63 features**, distilled to **12 ranked adoptions** + 6 rearchitecture themes + 11 anti-patterns; all findings persisted to `memory/zo-platform/research/2026-08-12-repo-reviews/` (per-repo markdown + `raw-findings.json`). **Sam decided: adopt all 12.** Work organized into **five layer-based workstreams** (A enforcement plane, B control plane, C execution substrate, D self-learning/platform oracle, E operator experience) — NOT source-repo categories, because features from different repos interlock into single mechanisms. Shipped this session: `plans/zo-v2-rearchitecture.md` (full plan: 6 gated phases, 20-check oracle, anti-scope; + `.gitignore` exception), `docs/reference/v2-rearchitecture.mdx` (all 12 features w/ provenance + repo links; added to mint.json Reference nav), `docs/roadmap.mdx` v2 section (4 pillars + repo credits), website §11 "What's next" (new section w/ 3 repo credit cards; quick start renumbered §12; drawer nav updated). **Verification caveat: no Node.js on this machine** — Astro build NOT run; website change verified via HTML-parser balance check + static-server DOM inspection (section text, all 4 links, drawer entry, renumbering all confirmed rendered). CI/deploy build must confirm. **Same session, part 2 — v2 Phase 1 (WS-A enforcement plane) SHIPPED** on branch `claude/v2-phase1-enforcement` (stacked on the plan branch, PR #106): (A1) `src/zo/contracts.py` — contracts.json emitted at decompose into memory_root (gate_mode precedent), `contract_produced` upgraded from prose placeholders to concrete paths (ownership ∩ required_artifacts, ownership-dir fallback), SubagentStop hook validates deliverables (missing/undersized/pattern/empty-dir) and blocks with a violation list; (A2) drift-guard Stop hook — completion-claim regex over the last assistant transcript message + added TODO/FIXME/NotImplementedError lines in `git diff HEAD` → block (env kill-switch `ZO_DRIFT_GUARD=0`); (A3) PreCompact (STATE flush + checkpoint decision), SessionEnd (summary backfill), PostToolUseFailure (`logs/comms/failures-{date}.jsonl` feed) — specs/memory.md recovery section updated to match (replaces the never-built periodic postToolUse checkpoint design); (A4) sealed-paths PreToolUse guard — memory-root control files (gate_mode/gate_nonce/gate_decision/contracts.json/sealed_paths) + user `sealed_paths` prefixes denied, off-limits write-scope enforced per contracts.json when agent identity present in hook input (plan check 6 AMENDED: no disallowedTools frontmatter exists for subagents and verifiers need scoped writes — path-scoped enforcement instead, fail-open without identity); (A5) nonce gates — minted at GATED (`secrets.token_hex(8)` → `gate_nonce` file), surfaced in `prepare_gate_review`, `apply_human_decision` raises PermissionError without it (single-use, cleared on terminal decisions), new `zo gates approve/reject --nonce` CLI writes DECISION_LOG + comms + `gate_decision` file consumed on next decompose, `/approve`+`/reject` slash commands rewritten to route through the CLI (forgeable hand-edit path CLOSED). All via one shim (`.claude/hooks/zo-hookkit.sh` → `python3 -m zo.hookkit`, venv-preferring, fail-open) + 6 new settings.json wirings. **First-ever hook-script tests** (subprocess + stdin JSON pattern). **854 → 904 passed / 7 skipped, ruff `src/` clean, validate-docs green.** 4 pre-existing integration tests updated to pass the nonce (designed behaviour change). Plan oracle checks 1-5 + 7 have passing seeded-failure tests; check 6 as amended. **Part 3 — live pre-PR verification (Sam-directed):** added always-on hook-trace observability (`logs/hook-trace-{date}.jsonl`, `ZO_HOOK_TRACE=0` off-switch) and verified in the live session itself: sealed-paths DENIED a real Write to gate_mode; drift-guard fired correctly-silent on a real Stop; subagent-stop fired with `agent_type`+`agent_id` in the live payload — **agent-identity open question RESOLVED** (per-agent enforcement keys correctly). Drift-guard now prefers the live payload's `last_assistant_message` (transcript parse = fallback). Caveats logged: PostToolUseFailure doesn't fire on nonzero-exit Bash (infrastructure errors only); PreCompact/SessionEnd not yet observed live; full `zo build` demo needs a machine with the claude CLI (this Mac has none — PR-046). **908 passed / 7 skipped, ruff clean.** **Next:** Phase 1 gate review → then WS-B control plane (plan-ledger.json) per `plans/zo-v2-rearchitecture.md`; prior Batch D/E + Tier-1 items queue behind v2. +**Session 040 (current) — pick up here.** Research + decision session: deep-dive review of three agent-orchestration repos (oh-my-claudecode, ruflo, ralph — cloned to `~/Documents/code/`) to inform the ZO v2 rearchitecture. 9-agent workflow (7 source-reading lenses + ZO baseline + adversarial synthesis, ~1.06M tokens) catalogued **63 features**, distilled to **12 ranked adoptions** + 6 rearchitecture themes + 11 anti-patterns; all findings persisted to `memory/zo-platform/research/2026-08-12-repo-reviews/` (per-repo markdown + `raw-findings.json`). **Sam decided: adopt all 12.** Work organized into **five layer-based workstreams** (A enforcement plane, B control plane, C execution substrate, D self-learning/platform oracle, E operator experience) — NOT source-repo categories, because features from different repos interlock into single mechanisms. Shipped this session: `plans/zo-v2-rearchitecture.md` (full plan: 6 gated phases, 20-check oracle, anti-scope; + `.gitignore` exception), `docs/reference/v2-rearchitecture.mdx` (all 12 features w/ provenance + repo links; added to mint.json Reference nav), `docs/roadmap.mdx` v2 section (4 pillars + repo credits), website §11 "What's next" (new section w/ 3 repo credit cards; quick start renumbered §12; drawer nav updated). **Verification caveat: no Node.js on this machine** — Astro build NOT run; website change verified via HTML-parser balance check + static-server DOM inspection (section text, all 4 links, drawer entry, renumbering all confirmed rendered). CI/deploy build must confirm. **Same session, part 2 — v2 Phase 1 (WS-A enforcement plane) SHIPPED** on branch `claude/v2-phase1-enforcement` (stacked on the plan branch, PR #106): (A1) `src/zo/contracts.py` — contracts.json emitted at decompose into memory_root (gate_mode precedent), `contract_produced` upgraded from prose placeholders to concrete paths (ownership ∩ required_artifacts, ownership-dir fallback), SubagentStop hook validates deliverables (missing/undersized/pattern/empty-dir) and blocks with a violation list; (A2) drift-guard Stop hook — completion-claim regex over the last assistant transcript message + added TODO/FIXME/NotImplementedError lines in `git diff HEAD` → block (env kill-switch `ZO_DRIFT_GUARD=0`); (A3) PreCompact (STATE flush + checkpoint decision), SessionEnd (summary backfill), PostToolUseFailure (`logs/comms/failures-{date}.jsonl` feed) — specs/memory.md recovery section updated to match (replaces the never-built periodic postToolUse checkpoint design); (A4) sealed-paths PreToolUse guard — memory-root control files (gate_mode/gate_nonce/gate_decision/contracts.json/sealed_paths) + user `sealed_paths` prefixes denied, off-limits write-scope enforced per contracts.json when agent identity present in hook input (plan check 6 AMENDED: no disallowedTools frontmatter exists for subagents and verifiers need scoped writes — path-scoped enforcement instead, fail-open without identity); (A5) nonce gates — minted at GATED (`secrets.token_hex(8)` → `gate_nonce` file), surfaced in `prepare_gate_review`, `apply_human_decision` raises PermissionError without it (single-use, cleared on terminal decisions), new `zo gates approve/reject --nonce` CLI writes DECISION_LOG + comms + `gate_decision` file consumed on next decompose, `/approve`+`/reject` slash commands rewritten to route through the CLI (forgeable hand-edit path CLOSED). All via one shim (`.claude/hooks/zo-hookkit.sh` → `python3 -m zo.hookkit`, venv-preferring, fail-open) + 6 new settings.json wirings. **First-ever hook-script tests** (subprocess + stdin JSON pattern). **854 → 904 passed / 7 skipped, ruff `src/` clean, validate-docs green.** 4 pre-existing integration tests updated to pass the nonce (designed behaviour change). Plan oracle checks 1-5 + 7 have passing seeded-failure tests; check 6 as amended. **Part 3 — live pre-PR verification (Sam-directed):** added always-on hook-trace observability (`logs/hook-trace-{date}.jsonl`, `ZO_HOOK_TRACE=0` off-switch) and verified in the live session itself: sealed-paths DENIED a real Write to gate_mode; drift-guard fired correctly-silent on a real Stop; subagent-stop fired with `agent_type`+`agent_id` in the live payload — **agent-identity open question RESOLVED** (per-agent enforcement keys correctly). Drift-guard now prefers the live payload's `last_assistant_message` (transcript parse = fallback). Caveats logged: PostToolUseFailure doesn't fire on nonzero-exit Bash (infrastructure errors only); PreCompact/SessionEnd not yet observed live; full `zo build` demo needs a machine with the claude CLI (this Mac has none — PR-046). **908 passed / 7 skipped, ruff clean.** **Part 4 — Phases merged + Phase 2 (WS-B control plane) SHIPPED:** #106 + #107 merged to main (stack conflict resolved by merging main into the branch, branch side kept — main had nothing unique). Then WS-B on `claude/v2-phase2-control-plane`: `src/zo/ledger.py` (plan-ledger.json: per-subtask entries w/ synthesized criteria, merge-preserving regeneration, atomic writes, phase_status map); oracle-owned flips wired at the two verified-completion sites (automated gate + nonce-verified human PROCEED), resets on ITERATE/loop-CONTINUE, attempts on mark_subtask_complete, ledger sealed via `_SEALED_DEFAULTS`; `## Stories` plan section + sizing lint in validate_plan (fires only when stories declared — legacy plans untouched); `zo status` renders the control-plane table from the ledger (STATE.md = fallback/projection). Phase-1 hardenings from recon: `contracts.set_active_phase` now atomic; `zo build` exports ZO_MEMORY_ROOT/ZO_DELIVERY_ROOT/ZO_CONTRACTS_PATH so per-project sealing works in delivery sessions. Deferred to Phase 3 (documented): evaluate_loop_state ledger input, session-restore cutover (PR-036 precedence). **929 passed / 7 skipped, ruff clean, validate-docs green. Oracle checks 8-10 seeded tests pass.** **Next:** Phase 2 PR review/merge → Phase 3 (WS-C: watchdog, then fresh-context loop — demo validation needs the Linux box w/ claude CLI, PR-046). **Session 039 (prior).** Test-only change making the CLI command-set test plugin-tolerant: `tests/unit/test_cli.py::TestCliGroup::test_cli_group_has_all_commands` asserted strict equality on `cli.commands`, which contradicts the extension points ZO itself shipped in PR #99 — any downstream build installing a `zo.commands` entry-point plugin (the documented mechanism in `zo.extensions.load_cli_plugins`) adds commands and turned the core suite red in that environment, despite zero core changes. Changed `assert expected == actual` → `assert expected <= actual` with an explanatory comment. Trade-off accepted: equality also caught *core* commands added without updating the test; subset still catches removals, and additions have always required updating this test's `expected` set anyway. Surfaced by the first real downstream plugin registering a new command group. No code, version, agents, or docs touched — single-assertion diff (+ memory). **854 passed / 7 skipped on Python 3.11 AND 3.12, ruff `src/` clean, validate-docs 0 failures.** **Next:** unchanged — Batch **D** / **E**, standing Tier-1 (caveman, onboarding). diff --git a/memory/zo-platform/sessions/session-040-2026-08-12.md b/memory/zo-platform/sessions/session-040-2026-08-12.md index 9aa71b0..7781f5e 100644 --- a/memory/zo-platform/sessions/session-040-2026-08-12.md +++ b/memory/zo-platform/sessions/session-040-2026-08-12.md @@ -75,10 +75,35 @@ Sam said go — WS-A shipped on `claude/v2-phase1-enforcement` (stacked): - 854 → 904 tests passed / 7 skipped; ruff clean; validate-docs green; first-ever hook-script tests (subprocess + stdin JSON) -## Next session - -Phase 1 gate review with Sam, then WS-B control plane (plan-ledger.json, -oracle-owned pass flags, sizing lint) per plans/zo-v2-rearchitecture.md -Phase 2. Watchdog (WS-C) after that. Note for Phase 6 e2e: confirm Claude -Code's SubagentStop/PreToolUse hook payloads carry agent identity in live -team sessions (enforcement is fail-open without it). +## Live verification + merges + Phase 2 (same session, parts 3-4) + +- Live pre-PR verification (Sam-directed): hook-trace observability + added; sealed-paths DENIED a real Write, drift-guard fired silent on a + real Stop, subagent-stop payload carries agent_type+agent_id (identity + question RESOLVED). PR-046: this Mac has no claude CLI/uv/npm. +- #106 (plan+docs+website) MERGED. #107 (Phase 1) MERGED after resolving + a stack conflict (merged main in, kept branch side). +- Phase 2 / WS-B SHIPPED as PR #108 (targets main, full CI GREEN): + src/zo/ledger.py, oracle-owned flips at the two verified-completion + sites, ledger sealed, `## Stories` + sizing lint, ledger-first + `zo status`; Phase-1 hardenings (atomic set_active_phase, hook env + exported by zo build). 929 passed / 7 skipped. + +## Next session — pick up here + +1. Confirm #108 merged (Phase 2 gate). If not, merge it (CI is green). +2. Start Phase 3 = WS-C on a fresh branch off main: + `claude/v2-phase3-substrate`. Watchdog FIRST (heartbeat JSON per + agent, external checker in LifecycleWrapper poll loop, never-block + taxonomy: context-limit/rate-limit/auth/user-abort, bounded nudges, + rate-limit wait-and-resume, PID+start-time identity) — oracle checks + 11-12, buildable/testable on this Mac. THEN the fresh-context loop + for ML Phase 4 (experiment_loop.py spawns fresh builder per iteration + from ledger+lineage+priors digest) — its go/no-go check 13 needs the + Linux box with the claude CLI (PR-046). +3. Deferred-into-Phase-3 items: evaluate_loop_state ledger input; + session-restore cutover from STATE.md to ledger (keep PR-036 GATED + precedence). PostToolUseFailure only catches infra errors → WS-D. +4. Playbook that worked: recon swarm (4-7 read-only mappers w/ exact + file:line integration points) → build → seeded-failure test per + mechanism + wiring test → memory protocol → commit → PR. diff --git a/specs/plan.md b/specs/plan.md index a5d423b..f565fff 100644 --- a/specs/plan.md +++ b/specs/plan.md @@ -251,6 +251,33 @@ These sections are included based on project needs. Milestones are informational. Agents do not skip work to meet deadlines. If a milestone is missed, the orchestrator logs the delay and notifies the human. +### 10. Stories (optional, v2 WS-B) + +An optional `## Stories` section declares explicit user stories with per-story +acceptance criteria. Each `### ` sub-heading is one story; bullets under +an `**Acceptance criteria:**` label become its criteria: + +```markdown +## Stories + +### Load and validate the dataset + +**Acceptance criteria:** +- `pytest tests/test_ingest.py` passes with exit code 0 +- data/processed/clean.csv exists +``` + +**Sizing lint (ralph's "Number One Rule"):** when stories are declared, every +story must carry at least one *machine-verifiable* acceptance criterion — a +threshold comparison (`RMSE <= 0.05`), an artifact path, or a backticked +runnable command. `validate_plan` rejects vague criteria ("works well") as an +error, which also fails `zo preflight` and `zo validate`. Plans without a +Stories section are unaffected. Story sizing guidance: each story should be +completable in a single agent context window. + +Parsed stories feed `plan-ledger.json` (the machine-readable progress ledger +whose `passes` flags only the oracle-verified gate paths may flip). + ### 10. Delivery Specification ```markdown diff --git a/specs/workflow.md b/specs/workflow.md index 06a8218..3b9dd3b 100644 --- a/specs/workflow.md +++ b/specs/workflow.md @@ -878,7 +878,7 @@ Within each sequential phase: 1. **Strict ordering:** Each subtask must complete fully before the next begins. No parallelization within a sequential phase. 2. **Dependency enforcement:** Downstream subtasks assume upstream artifacts are ready. Do not skip or reorder without explicit `plan.md` override. -3. **State tracking:** Orchestrator records subtask completion in `STATE.md` with timestamp and artifact location. +3. **State tracking:** the machine-readable record is `plan-ledger.json` in the project memory root (v2 WS-B) — one entry per subtask with acceptance criteria, attempts, and a `passes` flag only the orchestrator's oracle-verified gate paths may flip (the file is sealed against direct agent writes). `STATE.md` remains the human-readable projection and records subtask completion with timestamp and artifact location. 4. **Failure handling:** If a subtask fails: - Do NOT skip it or proceed to next subtask - Either retry the subtask, escalate to human with error details, or pause the phase diff --git a/src/zo/cli.py b/src/zo/cli.py index 51f0098..a405711 100644 --- a/src/zo/cli.py +++ b/src/zo/cli.py @@ -1056,6 +1056,12 @@ def build( extra_env["CLAUDE_AUTOCOMPACT_PCT_OVERRIDE"] = str( _LOW_TOKEN_PRESET["compact_threshold"], ) + # Enforcement-plane hook environment (WS-A/WS-B): the sealed-paths, + # contract, and ledger hooks resolve the project memory root and + # delivery repo from these — without them, delivery-repo sessions + # fall back to platform-memory defaults and per-project sealing + # silently no-ops (recon: sealing gap 1). + # Note: set after project context resolution below. # 2. Resolve project context (.zo/ or legacy) # If the plan lives inside a .zo/plans/ directory, infer delivery repo @@ -1067,6 +1073,9 @@ def build( target = ctx.make_target() memory = ctx.make_memory() memory.initialize_project() + extra_env["ZO_MEMORY_ROOT"] = str(memory.memory_root) + extra_env["ZO_DELIVERY_ROOT"] = str(target.target_repo) + extra_env["ZO_CONTRACTS_PATH"] = str(memory.memory_root / "contracts.json") # 3. Detect mode from state state_check = memory.read_state() @@ -2301,12 +2310,49 @@ def status(project_name: str | None, repo: str | None) -> None: state = memory.read_state() + # Control plane first (WS-B, check 8): when plan-ledger.json exists, + # progress renders from it — machine state, not STATE.md prose. + from zo.ledger import LEDGER_FILENAME, load_ledger, summarize + + ledger_doc = load_ledger(memory.memory_root / LEDGER_FILENAME) + current_phase = state.phase + if ledger_doc is not None and ledger_doc.entries: + counts = summarize(ledger_doc) + current_phase = next( + ( + p for p in ledger_doc.phase_status + if ledger_doc.phase_status[p] != "completed" + ), + current_phase, + ) + ledger_table = Table( + title="Control plane (plan-ledger.json — passes are oracle-owned)", + style=_AMBER, + ) + ledger_table.add_column("Phase", style="bold") + ledger_table.add_column("Status") + ledger_table.add_column("Passed") + ledger_table.add_column("Attempts") + ledger_table.add_column("Last failure") + for phase_id, status_str in ledger_doc.phase_status.items(): + passed, total = counts.get(phase_id, (0, 0)) + phase_entries = [ + e for e in ledger_doc.entries if e.phase_id == phase_id + ] + attempts = sum(e.attempts for e in phase_entries) + failures = [e.last_failure for e in phase_entries if e.last_failure] + ledger_table.add_row( + phase_id, status_str, f"{passed}/{total}", + str(attempts), (failures[-1][:60] if failures else "—"), + ) + console.print(ledger_table) + table = Table(title=f"Project: {project_name}", style=_AMBER) table.add_column("Field", style="bold") table.add_column("Value") table.add_row("Mode", state.mode) - table.add_row("Phase", state.phase) + table.add_row("Phase", current_phase) table.add_row("Last Subtask", state.last_completed_subtask or "none") table.add_row("Blockers", ", ".join(state.active_blockers) or "none") table.add_row("Next Steps", ", ".join(state.next_steps) or "none") diff --git a/src/zo/contracts.py b/src/zo/contracts.py index de6f51c..4915a18 100644 --- a/src/zo/contracts.py +++ b/src/zo/contracts.py @@ -170,13 +170,24 @@ def load_contracts(path: Path) -> ContractsFile | None: def set_active_phase(memory_root: Path, phase_id: str) -> None: - """Update ``active_phase`` in an existing contracts file, if present.""" + """Update ``active_phase`` in an existing contracts file, if present. + + Atomic (temp + ``os.replace``) — a torn read in the fail-open hook + layer would silently disable contract enforcement. + """ path = memory_root / CONTRACTS_FILENAME doc = load_contracts(path) if doc is None: return doc.active_phase = phase_id - path.write_text(doc.model_dump_json(indent=2), encoding="utf-8") + fd, tmp = tempfile.mkstemp(dir=str(memory_root), suffix=".tmp") + try: + with os.fdopen(fd, "w", encoding="utf-8") as fh: + fh.write(doc.model_dump_json(indent=2)) + os.replace(tmp, path) + finally: + if os.path.exists(tmp): + os.unlink(tmp) def _check_spec(repo_root: Path, spec: DeliverableSpec) -> str | None: diff --git a/src/zo/hookkit.py b/src/zo/hookkit.py index ced9b35..a574e3c 100644 --- a/src/zo/hookkit.py +++ b/src/zo/hookkit.py @@ -45,7 +45,8 @@ r"^\+.*(\bTODO\b|\bFIXME\b|\bXXX\b|NotImplementedError|raise NotImplemented\b)" ) _SEALED_DEFAULTS = ( - "gate_mode", "gate_nonce", "gate_decision", CONTRACTS_FILENAME, "sealed_paths", + "gate_mode", "gate_nonce", "gate_decision", CONTRACTS_FILENAME, + "plan-ledger.json", "sealed_paths", ) diff --git a/src/zo/ledger.py b/src/zo/ledger.py new file mode 100644 index 0000000..5656091 --- /dev/null +++ b/src/zo/ledger.py @@ -0,0 +1,254 @@ +"""Machine-readable plan ledger with oracle-owned pass flags (v2 WS-B). + +``plan-ledger.json`` lives in the project memory root beside ``gate_mode`` +and ``contracts.json``: one entry per (phase, subtask) with acceptance +criteria, a synthesized verification descriptor, and a boolean ``passes`` +that ONLY the orchestrator's oracle-verified code paths may flip — +builders' direct writes are denied by the sealed-paths hook (WS-A4). +"What is done" becomes a query over this file; STATE.md remains the +human-readable projection. + +Design notes: + - Regeneration merges: structure comes from the current workflow + decomposition, but ``passes``/``attempts``/``last_failure`` and + ``phase_status`` are preserved by ``subtask_id`` so a re-decompose + (plan edit, fresh session) never erases verified progress. + - Every write is atomic (temp file + ``os.replace``) — a torn ledger + read by a fail-open hook would silently unblock builders. + - Mutators are fail-open at the call site (the orchestrator wraps + them); a ledger IO problem must never crash a build. +""" + +from __future__ import annotations + +import os +import re +import tempfile +from datetime import UTC, datetime +from typing import TYPE_CHECKING + +from pydantic import BaseModel, Field + +if TYPE_CHECKING: + from collections.abc import Callable + from pathlib import Path + + from zo._orchestrator_models import WorkflowDecomposition + +__all__ = [ + "LEDGER_FILENAME", + "LedgerEntry", + "LedgerFile", + "emit_ledger", + "load_ledger", + "mark_phase_passed", + "record_attempt", + "record_phase_failure", + "reset_phase", + "set_phase_status", +] + +LEDGER_FILENAME = "plan-ledger.json" + + +class LedgerEntry(BaseModel): + """One subtask's machine-checkable progress record.""" + + subtask_id: str + phase_id: str + description: str + acceptance_criteria: list[str] = Field(default_factory=list) + verification: str = "" + passes: bool = False + attempts: int = 0 + last_failure: str | None = None + + +class LedgerFile(BaseModel): + """Top-level schema of ``plan-ledger.json``.""" + + version: int = 1 + project: str = "" + generated_at: datetime = Field(default_factory=lambda: datetime.now(UTC)) + phase_status: dict[str, str] = Field(default_factory=dict) + entries: list[LedgerEntry] = Field(default_factory=list) + + +def _slug(text: str) -> str: + return re.sub(r"[^a-z0-9]+", "-", text.lower()).strip("-") + + +def _atomic_write(path: Path, text: str) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + fd, tmp = tempfile.mkstemp(dir=str(path.parent), suffix=".tmp") + try: + with os.fdopen(fd, "w", encoding="utf-8") as fh: + fh.write(text) + os.replace(tmp, path) + finally: + if os.path.exists(tmp): + os.unlink(tmp) + + +def load_ledger(path: Path) -> LedgerFile | None: + """Load and parse the ledger; ``None`` on any problem (fail-open).""" + try: + return LedgerFile.model_validate_json(path.read_text(encoding="utf-8")) + except (OSError, ValueError): + return None + + +def _criteria_for_phase( + phase_id: str, + required_artifacts: list[str], + oracle_threshold: str | None, +) -> tuple[list[str], str]: + """Synthesize acceptance criteria + verification descriptor. + + Criteria come from what the platform can already mechanically check: + phase required-artifact existence (all phases) and the oracle + threshold (phase_4, where ``_finalize_experiments`` parses the + oracle's ``result.md``). Phases 1-3/5-6 are artifact-verified only — + stated explicitly rather than pretending otherwise. + """ + criteria = [f"artifact exists: {a}" for a in required_artifacts] + if phase_id == "phase_4" and oracle_threshold: + criteria.append(f"oracle threshold met: {oracle_threshold}") + verification = "oracle result.md tier evaluation (_finalize_experiments)" + elif required_artifacts: + verification = "artifact existence check (_check_artifacts)" + else: + verification = "phase gate evaluation" + return criteria, verification + + +def emit_ledger( + workflow: WorkflowDecomposition, + memory_root: Path, + project: str, + oracle_threshold: str | None = None, +) -> Path: + """Generate (or regenerate, merge-preserving) the plan ledger. + + Returns: + The path written. + """ + path = memory_root / LEDGER_FILENAME + previous = load_ledger(path) + prev_entries = ( + {e.subtask_id: e for e in previous.entries} if previous else {} + ) + prev_status = dict(previous.phase_status) if previous else {} + + entries: list[LedgerEntry] = [] + phase_status: dict[str, str] = {} + for phase in workflow.phases: + phase_status[phase.phase_id] = prev_status.get( + phase.phase_id, str(phase.status), + ) + criteria, verification = _criteria_for_phase( + phase.phase_id, phase.required_artifacts, oracle_threshold, + ) + for subtask in phase.subtasks: + subtask_id = f"{phase.phase_id}:{_slug(subtask)}" + old = prev_entries.get(subtask_id) + entries.append( + LedgerEntry( + subtask_id=subtask_id, + phase_id=phase.phase_id, + description=subtask, + acceptance_criteria=criteria, + verification=verification, + passes=old.passes if old else False, + attempts=old.attempts if old else 0, + last_failure=old.last_failure if old else None, + ) + ) + + doc = LedgerFile(project=project, phase_status=phase_status, entries=entries) + _atomic_write(path, doc.model_dump_json(indent=2)) + return path + + +def _mutate(memory_root: Path, fn: Callable[[LedgerFile], None]) -> bool: + """Load-modify-write the ledger atomically; False if absent/corrupt.""" + path = memory_root / LEDGER_FILENAME + doc = load_ledger(path) + if doc is None: + return False + fn(doc) + _atomic_write(path, doc.model_dump_json(indent=2)) + return True + + +def mark_phase_passed(memory_root: Path, phase_id: str) -> bool: + """Flip all of a phase's entries to ``passes: true``. + + ORACLE-OWNED: call only from the orchestrator's verified-completion + paths (automated gate with artifacts/oracle checks green, or a + nonce-verified human PROCEED). Builders cannot reach this — direct + ledger writes are sealed (WS-A4) and this module is not agent-facing. + """ + + def fn(doc: LedgerFile) -> None: + for entry in doc.entries: + if entry.phase_id == phase_id: + entry.passes = True + entry.last_failure = None + doc.phase_status[phase_id] = "completed" + + return _mutate(memory_root, fn) + + +def reset_phase(memory_root: Path, phase_id: str, reason: str) -> bool: + """Reset a phase's entries for rework (ITERATE / loop CONTINUE).""" + + def fn(doc: LedgerFile) -> None: + for entry in doc.entries: + if entry.phase_id == phase_id: + entry.passes = False + entry.last_failure = reason[:500] + doc.phase_status[phase_id] = "active" + + return _mutate(memory_root, fn) + + +def record_phase_failure(memory_root: Path, phase_id: str, reason: str) -> bool: + """Record a failed gate evaluation without resetting progress.""" + + def fn(doc: LedgerFile) -> None: + for entry in doc.entries: + if entry.phase_id == phase_id and not entry.passes: + entry.last_failure = reason[:500] + + return _mutate(memory_root, fn) + + +def record_attempt(memory_root: Path, phase_id: str, subtask: str) -> bool: + """Count a work attempt on a subtask (does NOT touch ``passes``).""" + subtask_id = f"{phase_id}:{_slug(subtask)}" + + def fn(doc: LedgerFile) -> None: + for entry in doc.entries: + if entry.subtask_id == subtask_id: + entry.attempts += 1 + + return _mutate(memory_root, fn) + + +def set_phase_status(memory_root: Path, phase_id: str, status: str) -> bool: + """Record a phase-status transition (active/gated/completed/blocked).""" + + def fn(doc: LedgerFile) -> None: + doc.phase_status[phase_id] = status + + return _mutate(memory_root, fn) + + +def summarize(doc: LedgerFile) -> dict[str, tuple[int, int]]: + """Per-phase (passed, total) counts for status rendering.""" + counts: dict[str, tuple[int, int]] = {} + for entry in doc.entries: + passed, total = counts.get(entry.phase_id, (0, 0)) + counts[entry.phase_id] = (passed + (1 if entry.passes else 0), total + 1) + return counts diff --git a/src/zo/orchestrator.py b/src/zo/orchestrator.py index ca656f0..a759087 100644 --- a/src/zo/orchestrator.py +++ b/src/zo/orchestrator.py @@ -17,6 +17,7 @@ from __future__ import annotations +import contextlib import hashlib import secrets from datetime import UTC, datetime @@ -25,6 +26,7 @@ from typing import TYPE_CHECKING from zo import contracts as zo_contracts +from zo import ledger as zo_ledger from zo._evolution_models import FailureRecord, FailureSeverity from zo._memory_models import ( Confidence, @@ -348,6 +350,7 @@ def decompose_plan(self) -> WorkflowDecomposition: self._restore_phase_states() self._consume_gate_decision() self._emit_contracts_file() + self._emit_plan_ledger() self._comms.log_decision( agent="orchestrator", title=f"Plan decomposed into {len(phases)} phases ({mode})", @@ -357,6 +360,46 @@ def decompose_plan(self) -> WorkflowDecomposition: self._session_state.phase = phases[0].phase_id return self._workflow + def _emit_plan_ledger(self) -> None: + """Generate/refresh ``plan-ledger.json`` (v2 WS-B, merge-preserving). + + Fail-open: emission problems are logged, never raised. + """ + if self._workflow is None: + return + threshold = ( + self._plan.oracle.target_threshold if self._plan.oracle else None + ) + try: + path = zo_ledger.emit_ledger( + self._workflow, + self._memory.memory_root, + self._plan.frontmatter.project_name, + oracle_threshold=threshold, + ) + except OSError as exc: + self._comms.log_error( + agent="orchestrator", + error_type="ledger_emission_failed", + severity="warning", + description=f"plan-ledger.json emission failed: {exc}", + ) + return + self._comms.log_decision( + agent="orchestrator", + title="Plan ledger emitted", + rationale=( + f"{sum(len(p.subtasks) for p in self._workflow.phases)} " + "subtask entries; passes flags are oracle-owned" + ), + outcome=str(path), confidence="high", + ) + + def _ledger_safe(self, fn: str, *args: object) -> None: + """Invoke a zo.ledger mutator, swallowing IO errors (fail-open).""" + with contextlib.suppress(OSError): + getattr(zo_ledger, fn)(self._memory.memory_root, *args) + def _consume_gate_decision(self) -> None: """Apply a nonce-verified CLI gate decision recorded while offline. @@ -746,6 +789,7 @@ def advance_phase(self, phase_id: str) -> GateEvaluation: zo_contracts.set_active_phase( self._memory.memory_root, phase_id, ) + self._ledger_safe("set_phase_status", phase_id, "gated") self._log_gate(ev) return ev @@ -762,6 +806,10 @@ def advance_phase(self, phase_id: str) -> GateEvaluation: f"{', '.join(all_missing)}" ), ) + self._ledger_safe( + "record_phase_failure", phase_id, + f"artifacts missing: {', '.join(all_missing)}", + ) self._log_gate(ev) return ev # For phase_4 (training/iteration), consult the autonomous @@ -775,6 +823,10 @@ def advance_phase(self, phase_id: str) -> GateEvaluation: return auto_iter phase.status = PhaseStatus.COMPLETED + # Oracle-owned flip (WS-B2): only this verified path — artifacts + # checked, experiments finalized, loop evaluator consulted — + # may set passes:true. Builders cannot write the ledger (sealed). + self._ledger_safe("mark_phase_passed", phase_id) self._generate_test_report(phase) self._generate_notebook(phase) self._generate_snapshot(phase, "automated", GateDecision.PROCEED) @@ -803,6 +855,7 @@ def mark_subtask_complete(self, phase_id: str, subtask: str) -> None: ) if subtask not in phase.completed_subtasks: phase.completed_subtasks.append(subtask) + self._ledger_safe("record_attempt", phase_id, subtask) if self._session_state is not None: self._session_state.last_completed_subtask = subtask self._memory.write_state(self._session_state) @@ -889,6 +942,8 @@ def apply_human_decision( self._memory.clear_gate_nonce() if decision == GateDecision.PROCEED: phase.status = PhaseStatus.COMPLETED + # Oracle-owned flip (WS-B2): nonce-verified human PROCEED. + self._ledger_safe("mark_phase_passed", phase_id) self._finalize_experiments(phase) self._generate_test_report(phase) self._generate_notebook(phase) @@ -896,9 +951,13 @@ def apply_human_decision( elif decision == GateDecision.ITERATE: phase.status = PhaseStatus.ACTIVE phase.completed_subtasks.clear() + self._ledger_safe( + "reset_phase", phase_id, notes or "human ITERATE decision", + ) self._abort_running_experiments(phase_id) elif decision == GateDecision.ESCALATE: phase.status = PhaseStatus.BLOCKED + self._ledger_safe("set_phase_status", phase_id, "blocked") else: phase.status = PhaseStatus.GATED @@ -1250,6 +1309,10 @@ def _auto_iterate_if_needed( # Continue — reset phase, next prompt mints child exp. phase.status = PhaseStatus.ACTIVE phase.completed_subtasks.clear() + self._ledger_safe( + "reset_phase", phase.phase_id, + f"loop CONTINUE: {decision.reason}", + ) return GateEvaluation( phase_id=phase.phase_id, gate_type=GateType.AUTOMATED, diff --git a/src/zo/plan.py b/src/zo/plan.py index 2535395..5c21531 100644 --- a/src/zo/plan.py +++ b/src/zo/plan.py @@ -150,6 +150,20 @@ class ExperimentLoopSpec(BaseModel): dead_end_threshold: float | None = None +class StorySpec(BaseModel): + """One user story from an optional ``## Stories`` section (v2 WS-B). + + Stories carry ralph's "Number One Rule": each must be sized to one + context window and carry machine-verifiable acceptance criteria — + a threshold comparison, an artifact path, or a runnable command. + ``validate_plan`` rejects stories whose criteria are none of these. + """ + + title: str + description: str = "" + acceptance_criteria: list[str] = Field(default_factory=list) + + class Plan(BaseModel): """The full parsed plan — top-level container.""" @@ -172,6 +186,10 @@ class Plan(BaseModel): # orchestrator when absent. Parsed from ``## Experiment Loop``. experiment_loop: ExperimentLoopSpec | None = None + # Optional per-story acceptance criteria (v2 WS-B). Parsed from + # ``## Stories``; the sizing lint fires only when stories exist. + stories: list[StorySpec] = Field(default_factory=list) + # Raw section map for introspection. raw_sections: dict[str, str] = Field(default_factory=dict) @@ -576,6 +594,8 @@ def _parse_agents(body: str) -> AgentConfig: "experiment loop": "experiment_loop", "experiment loop policy": "experiment_loop", "autonomous iteration": "experiment_loop", + "stories": "stories", + "user stories": "stories", } @@ -639,6 +659,7 @@ def parse_plan(path: Path) -> Plan: _parse_experiment_loop(mapped["experiment_loop"]) if "experiment_loop" in mapped else None ) + stories = _parse_stories(mapped["stories"]) if "stories" in mapped else [] return Plan( frontmatter=frontmatter, @@ -654,11 +675,65 @@ def parse_plan(path: Path) -> Plan: environment=mapped.get("environment"), open_questions=mapped.get("open_questions"), experiment_loop=experiment_loop, + stories=stories, raw_sections=raw_sections, source_path=path, ) +def _parse_stories(body: str) -> list[StorySpec]: + """Parse a ``## Stories`` section body into StorySpec entries. + + Each ``### <title>`` sub-heading starts a story; ``- `` bullets under + an ``Acceptance criteria:`` line (or all bullets, if no such label) + become its acceptance criteria; remaining prose is the description. + """ + stories: list[StorySpec] = [] + blocks = re.split(r"^###\s+", body, flags=re.MULTILINE) + for block in blocks[1:]: + lines = block.splitlines() + title = lines[0].strip() + rest = lines[1:] + criteria: list[str] = [] + prose: list[str] = [] + in_criteria = False + for line in rest: + stripped = line.strip() + if re.match(r"(?i)^\**acceptance criteria\**\s*:?\s*$", stripped): + in_criteria = True + continue + if stripped.startswith("- "): + criteria.append(stripped[2:].strip()) + elif stripped and not in_criteria: + prose.append(stripped) + stories.append( + StorySpec( + title=title, + description=" ".join(prose), + acceptance_criteria=criteria, + ) + ) + return stories + + +# Machine-verifiability heuristics for the sizing lint (v2 WS-B3): +# a criterion is machine-verifiable when it names a threshold comparison, +# an artifact path, or a runnable (backticked) command. +_CRITERION_THRESHOLD_RE = re.compile(r"[<>≤≥=]=?\s*\d|\d+(\.\d+)?\s*%") +_CRITERION_PATH_RE = re.compile( + r"\b[\w./-]+\.(py|md|json|yaml|yml|csv|txt|pt|onnx)\b|\b[\w-]+/[\w./-]+" +) +_CRITERION_COMMAND_RE = re.compile(r"`[^`]+`") + + +def _criterion_is_verifiable(criterion: str) -> bool: + return bool( + _CRITERION_THRESHOLD_RE.search(criterion) + or _CRITERION_PATH_RE.search(criterion) + or _CRITERION_COMMAND_RE.search(criterion) + ) + + _LOOP_FIELD_RE = re.compile( r"^\s*(max_iterations|plateau_epsilon|plateau_runs|stop_on_tier|dead_end_threshold)\s*:\s*(.+?)\s*$", re.MULTILINE, @@ -777,6 +852,33 @@ def validate_plan(plan: Plan) -> ValidationReport: message="At least one active agent must be specified.", )) + # --- Story sizing lint (v2 WS-B3, ralph's Number One Rule) --- + # Fires only when a ## Stories section is declared: every story must + # carry at least one machine-verifiable acceptance criterion (a + # threshold comparison, an artifact path, or a backticked command). + for story in plan.stories: + if not story.acceptance_criteria: + issues.append(ValidationIssue( + section="Stories", + message=( + f"Story '{story.title}' has no acceptance criteria. " + "Each story needs at least one machine-verifiable " + "criterion (threshold, artifact path, or `command`)." + ), + )) + elif not any( + _criterion_is_verifiable(c) for c in story.acceptance_criteria + ): + issues.append(ValidationIssue( + section="Stories", + message=( + f"Story '{story.title}' has no machine-verifiable " + "acceptance criterion — none contains a threshold " + "comparison, artifact path, or `command`. Vague " + "criteria cannot gate work (v2 sizing lint)." + ), + )) + return ValidationReport( valid=len(issues) == 0, issues=issues, diff --git a/tests/unit/test_cli.py b/tests/unit/test_cli.py index bd03a7a..edc68e7 100644 --- a/tests/unit/test_cli.py +++ b/tests/unit/test_cli.py @@ -843,6 +843,56 @@ def test_status_shows_state( assert "test-project" in result.output + + def test_status_renders_from_ledger( + self, runner: click.testing.CliRunner, tmp_path: Path + ) -> None: + """Check 8 (v2 WS-B): with plan-ledger.json present, zo status + renders progress from the control plane, not STATE.md prose.""" + from zo._orchestrator_models import ( + GateType, + PhaseDefinition, + WorkflowDecomposition, + ) + from zo.ledger import emit_ledger, mark_phase_passed + + mem_root = tmp_path / "memory" / "test-project" + mem_root.mkdir(parents=True) + (mem_root / "sessions").mkdir() + (mem_root / "STATE.md").write_text( + "---\ntimestamp: 2026-01-01T00:00:00Z\nmode: build\n" + "phase: stale-prose-phase\n---\n", + encoding="utf-8", + ) + wf = WorkflowDecomposition( + mode="classical_ml", + phases=[ + PhaseDefinition( + phase_id="phase_1", name="Data", description="d", + gate_type=GateType.BLOCKING, + subtasks=["Audit", "Schema"], + ), + PhaseDefinition( + phase_id="phase_2", name="Features", description="f", + gate_type=GateType.BLOCKING, subtasks=["Engineer"], + ), + ], + ) + emit_ledger(wf, mem_root, "test-project") + mark_phase_passed(mem_root, "phase_1") + + with patch("zo.cli._zo_root", return_value=tmp_path): + result = runner.invoke(cli, ["status", "test-project"]) + + assert result.exit_code == 0 + assert "Control plane" in result.output + assert "2/2" in result.output # phase_1 fully passed + assert "0/1" in result.output # phase_2 pending + # Phase shown comes from the ledger (first non-completed phase), + # not the stale STATE.md prose. + assert "phase_2" in result.output + + # --------------------------------------------------------------------------- # build command # --------------------------------------------------------------------------- diff --git a/tests/unit/test_gate_nonce.py b/tests/unit/test_gate_nonce.py index 5d55223..fd05e69 100644 --- a/tests/unit/test_gate_nonce.py +++ b/tests/unit/test_gate_nonce.py @@ -127,3 +127,42 @@ def test_stale_decision_for_unknown_phase_discarded(self, wired) -> None: memory.write_gate_decision("phase_99", "proceed", "stale") orch.decompose_plan() assert memory.read_gate_decision() is None + + +class TestLedgerOracleFlip: + """Check 9 (landing half): the nonce-verified gate path flips the ledger.""" + + def test_nonce_approval_marks_ledger_phase_passed(self, wired) -> None: + from zo.ledger import LEDGER_FILENAME, load_ledger, summarize + + orch, memory = wired + phase_id = _gate_first_phase(orch) + ledger = load_ledger(memory.memory_root / LEDGER_FILENAME) + assert ledger is not None # emitted at decompose + passed, total = summarize(ledger).get(phase_id, (0, 0)) + assert passed == 0 # nothing passes before the oracle path runs + + orch.apply_human_decision( + phase_id, GateDecision.PROCEED, "verified", + nonce=memory.read_gate_nonce(), + ) + ledger = load_ledger(memory.memory_root / LEDGER_FILENAME) + passed, total = summarize(ledger)[phase_id] + assert passed == total > 0 + assert ledger.phase_status[phase_id] == "completed" + + def test_iterate_resets_ledger(self, wired) -> None: + from zo.ledger import LEDGER_FILENAME, load_ledger, summarize + + orch, memory = wired + phase_id = _gate_first_phase(orch) + orch.apply_human_decision( + phase_id, GateDecision.ITERATE, "rework the audit", + nonce=memory.read_gate_nonce(), + ) + ledger = load_ledger(memory.memory_root / LEDGER_FILENAME) + passed, _ = summarize(ledger)[phase_id] + assert passed == 0 + assert ledger.phase_status[phase_id] == "active" + entry = next(e for e in ledger.entries if e.phase_id == phase_id) + assert "rework the audit" in (entry.last_failure or "") diff --git a/tests/unit/test_ledger.py b/tests/unit/test_ledger.py new file mode 100644 index 0000000..d3b2558 --- /dev/null +++ b/tests/unit/test_ledger.py @@ -0,0 +1,165 @@ +"""Tests for zo.ledger — the WS-B control plane (plan oracle checks 8-9). + +Seeded-failure pattern: builders' direct ledger writes are denied by the +sealed-paths hook; only the orchestrator's oracle-verified paths flip +``passes: true``. +""" + +from __future__ import annotations + +import io +import json +from typing import TYPE_CHECKING + +from zo import hookkit +from zo._orchestrator_models import ( + GateType, + PhaseDefinition, + WorkflowDecomposition, +) +from zo.ledger import ( + LEDGER_FILENAME, + emit_ledger, + load_ledger, + mark_phase_passed, + record_attempt, + record_phase_failure, + reset_phase, + set_phase_status, + summarize, +) + +if TYPE_CHECKING: + from pathlib import Path + + +def _workflow() -> WorkflowDecomposition: + return WorkflowDecomposition( + mode="classical_ml", + phases=[ + PhaseDefinition( + phase_id="phase_1", name="Data Review", description="d", + gate_type=GateType.BLOCKING, + subtasks=["Raw data audit", "Schema check"], + required_artifacts=["data/reports/data_quality.md"], + ), + PhaseDefinition( + phase_id="phase_4", name="Training", description="t", + gate_type=GateType.AUTOMATED, + subtasks=["Train baseline"], + ), + ], + ) + + +class TestGeneration: + def test_entries_per_subtask_with_synthesized_criteria(self, tmp_path: Path): + path = emit_ledger(_workflow(), tmp_path, "demo", "RMSE <= 0.05") + doc = load_ledger(path) + assert doc is not None + assert len(doc.entries) == 3 + audit = next(e for e in doc.entries if "raw-data-audit" in e.subtask_id) + assert audit.passes is False + assert audit.acceptance_criteria == [ + "artifact exists: data/reports/data_quality.md" + ] + train = next(e for e in doc.entries if e.phase_id == "phase_4") + assert "oracle threshold met: RMSE <= 0.05" in train.acceptance_criteria + assert "oracle" in train.verification + + def test_phase_status_initialized(self, tmp_path: Path): + doc = load_ledger(emit_ledger(_workflow(), tmp_path, "demo")) + assert set(doc.phase_status) == {"phase_1", "phase_4"} + + def test_regeneration_preserves_progress(self, tmp_path: Path): + emit_ledger(_workflow(), tmp_path, "demo") + mark_phase_passed(tmp_path, "phase_1") + record_attempt(tmp_path, "phase_4", "Train baseline") + # Re-decompose (plan edit / fresh session) regenerates structure... + doc = load_ledger(emit_ledger(_workflow(), tmp_path, "demo")) + # ...but verified progress and bookkeeping survive. + assert all(e.passes for e in doc.entries if e.phase_id == "phase_1") + train = next(e for e in doc.entries if e.phase_id == "phase_4") + assert train.attempts == 1 + assert doc.phase_status["phase_1"] == "completed" + + +class TestMutators: + def test_mark_phase_passed_scopes_to_phase(self, tmp_path: Path): + emit_ledger(_workflow(), tmp_path, "demo") + mark_phase_passed(tmp_path, "phase_1") + doc = load_ledger(tmp_path / LEDGER_FILENAME) + assert summarize(doc)["phase_1"] == (2, 2) + assert summarize(doc)["phase_4"] == (0, 1) + + def test_reset_phase_clears_passes_and_records_reason(self, tmp_path: Path): + emit_ledger(_workflow(), tmp_path, "demo") + mark_phase_passed(tmp_path, "phase_1") + reset_phase(tmp_path, "phase_1", "human ITERATE: rework features") + doc = load_ledger(tmp_path / LEDGER_FILENAME) + entry = next(e for e in doc.entries if e.phase_id == "phase_1") + assert entry.passes is False + assert "rework features" in entry.last_failure + assert doc.phase_status["phase_1"] == "active" + + def test_record_phase_failure_keeps_passes(self, tmp_path: Path): + emit_ledger(_workflow(), tmp_path, "demo") + record_phase_failure(tmp_path, "phase_1", "artifacts missing: x.md") + doc = load_ledger(tmp_path / LEDGER_FILENAME) + entry = next(e for e in doc.entries if e.phase_id == "phase_1") + assert entry.passes is False + assert "artifacts missing" in entry.last_failure + + def test_attempts_do_not_touch_passes(self, tmp_path: Path): + emit_ledger(_workflow(), tmp_path, "demo") + record_attempt(tmp_path, "phase_1", "Raw data audit") + record_attempt(tmp_path, "phase_1", "Raw data audit") + doc = load_ledger(tmp_path / LEDGER_FILENAME) + audit = next(e for e in doc.entries if "raw-data-audit" in e.subtask_id) + assert audit.attempts == 2 + assert audit.passes is False + + def test_set_phase_status(self, tmp_path: Path): + emit_ledger(_workflow(), tmp_path, "demo") + set_phase_status(tmp_path, "phase_1", "gated") + assert load_ledger(tmp_path / LEDGER_FILENAME).phase_status["phase_1"] == "gated" + + def test_mutators_fail_open_without_ledger(self, tmp_path: Path): + assert mark_phase_passed(tmp_path, "phase_1") is False + assert reset_phase(tmp_path, "phase_1", "x") is False + + def test_corrupt_ledger_fails_open(self, tmp_path: Path): + (tmp_path / LEDGER_FILENAME).write_text("{broken") + assert load_ledger(tmp_path / LEDGER_FILENAME) is None + assert mark_phase_passed(tmp_path, "phase_1") is False + + +class TestOracleOwnership: + """Plan oracle check 9: builder flip blocked, oracle flip lands.""" + + def test_builder_direct_write_denied_by_sealed_paths( + self, tmp_path: Path, monkeypatch, capsys, + ): + """A builder agent's Write to plan-ledger.json is denied (seeded).""" + mem = tmp_path / "mem" + emit_ledger(_workflow(), mem, "demo") + monkeypatch.setenv("ZO_REPO_ROOT", str(tmp_path)) + monkeypatch.setenv("ZO_MEMORY_ROOT", str(mem)) + monkeypatch.setattr( + "sys.stdin", + io.StringIO(json.dumps({ + "agent_name": "model-builder", + "tool_input": {"file_path": str(mem / LEDGER_FILENAME)}, + })), + ) + assert hookkit.main(["sealed-paths"]) == 0 + out = json.loads(capsys.readouterr().out) + assert out["hookSpecificOutput"]["permissionDecision"] == "deny" + + def test_oracle_flip_lands_through_module_api(self, tmp_path: Path): + """The orchestrator's verified path (mark_phase_passed) succeeds — + it runs as platform code, outside the agent tool surface.""" + emit_ledger(_workflow(), tmp_path, "demo") + assert mark_phase_passed(tmp_path, "phase_1") is True + doc = load_ledger(tmp_path / LEDGER_FILENAME) + assert summarize(doc)["phase_1"] == (2, 2) diff --git a/tests/unit/test_plan.py b/tests/unit/test_plan.py index 620bfdd..9afaeed 100644 --- a/tests/unit/test_plan.py +++ b/tests/unit/test_plan.py @@ -679,3 +679,98 @@ def test_empty_adaptation_body_is_skipped(self, tmp_path: Path) -> None: assert "xai-agent" not in names # empty body skipped assert "domain-evaluator" in names assert plan.workflow.mode == WorkflowMode.CLASSICAL_ML + + +# --------------------------------------------------------------------------- +# Stories & sizing lint (v2 WS-B3, plan oracle check 10) +# --------------------------------------------------------------------------- + + +STORIES_SECTION_VALID = """ +## Stories + +### Load and validate the dataset + +Ingest the raw CSV and produce a validated frame. + +**Acceptance criteria:** +- `pytest tests/test_ingest.py` passes with exit code 0 +- data/processed/clean.csv exists + +### Train the baseline + +**Acceptance criteria:** +- Validation RMSE <= 0.05 on the held-out split +""" + +STORIES_SECTION_VAGUE = """ +## Stories + +### Make the model good + +**Acceptance criteria:** +- The model works well and the code is clean +""" + +STORIES_SECTION_EMPTY_CRITERIA = """ +## Stories + +### A story with no criteria + +Just prose, no bullets at all. +""" + + +class TestStoriesParsing: + def test_valid_stories_parse(self, tmp_path: Path) -> None: + plan = parse_plan( + _write_plan(tmp_path, MINIMAL_VALID_PLAN + STORIES_SECTION_VALID) + ) + assert len(plan.stories) == 2 + first = plan.stories[0] + assert first.title == "Load and validate the dataset" + assert "validated frame" in first.description + assert len(first.acceptance_criteria) == 2 + + def test_plan_without_stories_has_empty_list(self, tmp_path: Path) -> None: + plan = parse_plan(_write_plan(tmp_path, MINIMAL_VALID_PLAN)) + assert plan.stories == [] + + +class TestSizingLint: + """Plan oracle check 10: non-verifiable stories are rejected.""" + + def test_machine_verifiable_stories_pass(self, tmp_path: Path) -> None: + plan = parse_plan( + _write_plan(tmp_path, MINIMAL_VALID_PLAN + STORIES_SECTION_VALID) + ) + report = validate_plan(plan) + assert not [i for i in report.issues if i.section == "Stories"] + + def test_vague_story_rejected(self, tmp_path: Path) -> None: + """The seeded violation: criteria with no threshold/path/command.""" + plan = parse_plan( + _write_plan(tmp_path, MINIMAL_VALID_PLAN + STORIES_SECTION_VAGUE) + ) + report = validate_plan(plan) + story_issues = [i for i in report.issues if i.section == "Stories"] + assert len(story_issues) == 1 + assert "machine-verifiable" in story_issues[0].message + assert report.valid is False + + def test_story_without_criteria_rejected(self, tmp_path: Path) -> None: + plan = parse_plan( + _write_plan( + tmp_path, MINIMAL_VALID_PLAN + STORIES_SECTION_EMPTY_CRITERIA + ) + ) + report = validate_plan(plan) + story_issues = [i for i in report.issues if i.section == "Stories"] + assert len(story_issues) == 1 + assert "no acceptance criteria" in story_issues[0].message + + def test_lint_silent_without_stories_section(self, tmp_path: Path) -> None: + """Legacy plans without ## Stories are untouched by the lint.""" + plan = parse_plan(_write_plan(tmp_path, MINIMAL_VALID_PLAN)) + report = validate_plan(plan) + assert not [i for i in report.issues if i.section == "Stories"]