From addcdc633aa89b9e208fdcd6a89772cbcefd5266 Mon Sep 17 00:00:00 2001 From: Alberto Serrano-Calva Date: Mon, 27 Jul 2026 19:39:20 -0400 Subject: [PATCH 1/3] feat(bp-137): P1-P5 eligibility as a conjunctive three-valued predicate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit dn-autopilot-and-delegated-blessing §2.4 defines low-stakes work structurally and §2.8 makes those five predicates *the* reversibility guarantee. This computes them, and refuses on ABSENCE exactly as it refuses on VIOLATION. Two-valued conjunction is where the vacuous pass lives: "every glob resolves inside the worktree" is true over an empty write_scope, and "every item carries the flag as no" is true over a plan with no items. So a term the input did not answer returns UNDETERMINED, and UNDETERMINED is absorbing (invariant 7). `conjoin` takes a keyed mapping and requires the complete {P1..P5} key set, so `all(...)` has no empty-sequence overload to be vacuously true on; a test enumerates all 3**5 = 243 combinations and asserts exactly one passes. P3 carries the finding-0263 correction: §2.4 names a machine-readable per-item field that does not exist, so a P3 built to the note's text passes on every plan in the repository. Implemented instead as a pinned regex over the §7 item body whose value must normalize to exactly `no`; bp-137 §6 is authoritative until a superseding note says otherwise. The banner is sliced out of __doc__ rather than duplicated, and `report` emits it so the caveat reaches the owner's phone, not just a builder reading the source. P4's lexical blind spot and P5's uncheckable runtime half are named in the same output. Glob math is _lib.matches_any — the matcher scope-guard itself runs, asserted by identity and by an AST test that no local glob function exists. Two matchers that disagree would bless a scope the guard reads differently. Mutation campaign: 12 mutants, 12 killed, 0 survivors — including the load-bearing `all(t is not Result.FAIL)`, dropping the fence scan, and returning PASS on an empty scope. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01GTYQT2QsWskJW4HmgobyrV --- scripts/autopilot_eligibility.py | 599 +++++++++++++++++ tests/unit/test_autopilot_eligibility.py | 796 +++++++++++++++++++++++ 2 files changed, 1395 insertions(+) create mode 100644 scripts/autopilot_eligibility.py create mode 100644 tests/unit/test_autopilot_eligibility.py diff --git a/scripts/autopilot_eligibility.py b/scripts/autopilot_eligibility.py new file mode 100644 index 0000000..13629ff --- /dev/null +++ b/scripts/autopilot_eligibility.py @@ -0,0 +1,599 @@ +#!/usr/bin/env python +"""Autopilot eligibility — §2.4's P1–P5 as a **conjunctive three-valued predicate** (bp-137). + +`dn-autopilot-and-delegated-blessing` §2.4 defines low-stakes work structurally: *"work is +low-stakes iff its complete rollback is a git operation"*, decided by five conjunctive +predicates over a plan's own fields. §2.8 makes those five *the* reversibility guarantee — +P1–P4 jointly mean every effect of a run is uncommitted-to-main, git-tracked, +stored-data-free and live-state-free. This module computes them. + + uv run scripts/autopilot_eligibility.py check + -> exit 0 iff all five predicates PASS; exit 1 otherwise, with one diagnostic line + per non-PASS term naming the predicate, the result, and the evidence + uv run scripts/autopilot_eligibility.py report + -> prints the capsule-ready one-line `achievable:` block on stdout (the correction + banner below goes to stderr, so stdout stays a single embeddable line); + exit code as for `check` + +WHAT THIS TOOL IS NOT. It computes a **necessary** condition and holds no authority. The +sufficient condition is the owner issuing a code from his phone (§2.4: *"The agent's role in +eligibility is exactly nothing"*). No code path here represents a grant, reads a secret, or +touches a `status:` line; the foundation denylist (`CONSTITUTION.md`, `eval/golden/**`, +`eval/golden.py`) binds beneath every grant regardless of what this returns (invariant 5). + +THE THREE-VALUED DOMAIN, and why. Two-valued conjunction is where the vacuous pass lives: +"every glob resolves inside the worktree" is *true* over an empty `write_scope`, and "every +item carries the flag as no" is *true* over a plan with no items. Both are the false-success +shape (`docs/brainstorms/the-false-success-rule.md`): the observable the check consumes is +not causally downstream of the property claimed. So a term that the input did not answer +returns `UNDETERMINED`, and `UNDETERMINED` is **absorbing** under the conjunction — +invariant 7 (*"Ambiguity … always resolves toward halting"*) applied at the gate's mouth. +`conjoin` takes a **keyed mapping and requires the complete term set**, so there is no +overload on which `all(...)` can be vacuously true; overall PASS is unreachable while any +term is not PASS, and a test enumerates all 3**5 = 243 combinations to prove it. + +REUSE, not re-derivation. The glob math is `_lib.matches_any` — the *same* matcher +`scope-guard` runs (`.claude/hooks/_lib.py:150-177`). A predicate with its own glob +semantics would bless a scope the guard reads differently; that is a security-relevant +duplication, not a style one. `_lib` is imported here and never edited (plan §9 non-goal 3). + +Stdlib only otherwise — no `os`, no `subprocess`, no `core`, no `config`, so there is +structurally no route to an environment variable, a Keychain item or any other secret +(`finding-0207`'s constraint, held by construction; the AST is asserted in +`tests/unit/test_autopilot_eligibility.py`). It never writes: every path is read-only. + +--- correction banner (finding-0263) --- + +§2.4's P3 reads *"every plan item carries `touches_stored_data: false`"* — a machine-readable +per-item field that DOES NOT EXIST. Measured over `docs/build-plans/*/plan.md`: the plans +carry the flag as prose in at least twenty spellings and the literal string +`touches_stored_data: false` appears in ZERO of them. A P3 built to the note's text therefore +returns PASS on every plan in the repository, including one that rewrites the vector store. + +The check implemented here reads the §7 item BODY with a pinned regex and requires the +captured value, normalized, to be exactly `no`. Every hedge — `no (reads the corpus)`, +`reads only` — is a FAIL. An absent or duplicated flag line, and a plan with zero +`### Item ` headings, are UNDETERMINED. + +`docs/build-plans/bp-137/plan.md` §6 is the AUTHORITATIVE form until a superseding design +note says otherwise. `dn-autopilot-and-delegated-blessing` is ratified and agent-immutable +(A8), so the correction lives here rather than in the note. Any divergence between §6 and a +future note is a `spec-defect`, never a silent re-interpretation. + +P1 and P2 read the plan's DECLARED `write_scope`, which is not the effective allow-set +`scope-guard` enforces: `cmd_scope_check` (`.claude/hooks/_lib.py:464-466`) widens it with +the plan file, the plan's `journal.md` and `docs/findings/**`. Harmless for P2 — findings are +not enforcement surfaces — but the asymmetry is real and is stated rather than dropped. + +P4 is a LEXICAL scan with a pinned deny-list over §7's prose, fenced blocks included (a +command in a fence is still a command). It cannot see intent: an external call reached +through a helper, or a `deploy` spelled differently, passes it. It is conservative by +construction and will fire on a plan that merely *mentions* a token — including a non-goal +that promises never to run one. Announced, not elided; plan §11 row 2 rejects tuning it with +prose-context heuristics. + +P5 checks the plan's declared bound only. Un-self-extendability is a property of the runtime, +not of the plan text, and is NOT checked here. + +--- end correction banner --- +""" + +from __future__ import annotations + +import argparse +import enum +import re +import sys +from dataclasses import dataclass, field +from pathlib import Path + +ROOT = Path(__file__).resolve().parent.parent + +# Reuse the guard's own matcher and front-matter parser — never re-derive them (plan §2 DRY +# audit; the `scripts/board.py:33-38` idiom). +sys.path.insert(0, str(ROOT / ".claude" / "hooks")) +from _lib import matches_any, parse_front_matter # type: ignore[import-not-found] # noqa: E402 + +_BANNER_START = "--- correction banner (finding-0263) ---" +_BANNER_END = "--- end correction banner ---" + + +def _correction_banner() -> str: + """Slice the correction banner out of this module's own docstring. + + Sliced rather than duplicated into a literal so the docstring a builder reads and the + caveat the owner reads cannot drift (owner DRY rule). It raises rather than returning + `""` when the markers are gone: a report that silently loses its caveat is precisely the + false statement §2.4's *"printed in the capsule the owner reads"* makes dangerous. + """ + doc = __doc__ or "" + start = doc.find(_BANNER_START) + end = doc.find(_BANNER_END, start + 1) + if start == -1 or end == -1: + raise RuntimeError( + "the finding-0263 correction banner is missing from the module docstring; " + "this tool must not report an eligibility verdict without it" + ) + return doc[start + len(_BANNER_START) : end].strip() + + +P3_CORRECTION_BANNER = _correction_banner() + + +class Result(enum.Enum): + """A predicate's verdict. `UNDETERMINED` means *the input did not answer the question* — + it is not a weak PASS, and it is absorbing under `conjoin`.""" + + PASS = "pass" + FAIL = "fail" + UNDETERMINED = "undetermined" + + +@dataclass(frozen=True) +class Term: + """One of P1–P5: its verdict plus the evidence a diagnostic line prints.""" + + name: str + result: Result + evidence: str + + +#: The complete term set. `conjoin` requires exactly these keys — see its docstring. +TERM_NAMES: tuple[str, ...] = ("P1", "P2", "P3", "P4", "P5") + +#: §2.4's P2 forbidden set, verbatim (`dn-autopilot-and-delegated-blessing.md:320`). Broader +#: than the foundation denylist, which binds beneath every grant regardless (invariant 5) and +#: is deliberately NOT re-implemented here. +FORBIDDEN_SCOPE: tuple[str, ...] = ( + "CLAUDE.md", + ".claude/hooks/**", + ".claude/settings.json", + "docs/design-notes/**", + "eval/**", +) + +#: Concrete paths inside the forbidden set, used for the *covering* direction of P2: a scope +#: glob like `.claude/**`, `docs/**` or `**` names no forbidden pattern but reaches one. +FORBIDDEN_WITNESSES: tuple[str, ...] = ( + "CLAUDE.md", + ".claude/hooks/_lib.py", + ".claude/hooks/scope-guard.sh", + ".claude/settings.json", + "docs/design-notes/agent-workflow.md", + "eval/golden.py", + "eval/metrics.py", +) + +#: P4's pinned deny-list (plan §7 Item 16), `(label, pattern)`, matched case-insensitively +#: over §7's prose INCLUDING fenced blocks. Conservative by construction: `deploy` also fires +#: on "deployment" and on a non-goal that merely says "this plan never runs `deploy`". That +#: is a stated property, not a bug — plan §11 row 2 rejects prose-context heuristics. +LIVE_STATE_TOKENS: tuple[tuple[str, str], ...] = ( + ("deploy", r"deploy"), + ("palace lifecycle mutation", r"palace\s+(?:start|stop|restart|up|down)\b"), + ("keychain read", r"security\s+find-generic-password"), + ("secret read", r"get_secret"), + ("aws sdk", r"\bboto3\b"), + ("aws cli", r"\baws\s"), + ("network call", r"\bcurl\b"), + ("shell-out", r"\bsubprocess\b"), +) + +_SECTION_7 = re.compile(r"^##[ \t]*7[.)]?[ \t]", re.M) +_ANY_SECTION = re.compile(r"^##[ \t]", re.M) +_ITEM_HEADING = re.compile(r"^###[ \t]+Item[ \t]", re.M) + +#: P3's pinned regex (plan §6, the finding-0263 correction). The value is what follows the +#: bolded question on the same line; §7 items in this repo carry it as a bullet. +_FLAG_LINE = re.compile( + r"^[ \t]*[-*][ \t]*\*\*Touches stored data\?\*\*[ \t]*(?P.*)$", re.M +) + +#: The flag line commonly runs on into the next bolded field (57 of the measured 111 plans), +#: so the captured value is truncated at whichever of these appears first. +_TRUNCATE_AT: tuple[str, ...] = ("**Parallelizable?**", "**Depends on:**") + +#: The standing caveats carried in the one-line report. The full statements are in +#: `P3_CORRECTION_BANNER`, which `report` writes to stderr; this is the phone-sized pointer. +REPORT_CAVEATS = "P3=finding-0263 form; P4 lexical, blind to intent; P5 plan-side bound only" + +_ABSENT = ("", "null", "none", "~") + + +def _is_absent(value: object) -> bool: + """`_lib.parse_front_matter` does not interpret scalars, so a YAML `null` arrives as the + literal string `"null"` (the `scripts/board.py:58-67` observation). Absent, empty and + `null` are the same thing here: **undetermined**.""" + if value is None: + return True + if isinstance(value, str): + return value.strip().lower() in _ABSENT + if isinstance(value, (list, tuple)): + return len(value) == 0 + return False + + +def write_scope(front_matter: dict[str, object]) -> list[str]: + """The plan's declared capability, as `_lib.plan_write_scope` reads it. + + ⚑ NOT the effective allow-set `scope-guard` enforces: `cmd_scope_check` + (`.claude/hooks/_lib.py:464-466`) widens it with the plan file, the plan's `journal.md` + and `docs/findings/**`. That asymmetry is stated in the report rather than silently + dropped — it is harmless for P2 (findings are not enforcement surfaces) but it is real. + """ + raw = front_matter.get("write_scope") + if isinstance(raw, list): + return [str(x).strip() for x in raw if str(x).strip()] + if isinstance(raw, str) and not _is_absent(raw): + return [raw.strip()] + return [] + + +def deglue(entry: str) -> tuple[str, bool]: + """Split a `write_scope` entry from an inline `#` comment glued to it. + + `_lib._scalar` deliberately leaves a `#` intact on an UNQUOTED scalar (`:218-233`), so + `- eval/metrics.py # absorbed` reaches the guard as the *glob* `eval/metrics.py # + absorbed`, which matches nothing — the `bp-066` / `finding-0085` footgun. Returns the + intended glob and whether a comment was glued to it. + """ + idx = entry.find("#") + if idx == -1: + return entry.strip(), False + return entry[:idx].strip(), True + + +def escapes_root(glob: str) -> bool: + """True if the glob cannot resolve inside the worktree: absolute, home-relative, a drive + letter, empty, or a `..` chain that walks above the repo root. Pure string arithmetic — + no filesystem, no `os`.""" + g = glob.replace("\\", "/").strip() + if not g: + return True + if g.startswith("/") or g.startswith("~"): + return True + if re.match(r"^[A-Za-z]:", g): + return True + depth = 0 + for seg in g.split("/"): + if seg in ("", "."): + continue + if seg == "..": + depth -= 1 + if depth < 0: + return True + else: + depth += 1 + return False + + +def check_p1(scope: list[str]) -> Term: + """P1 — repo-confined: every `write_scope` glob resolves inside the worktree. + + ⚑ Degenerate input: an empty or absent `write_scope`. "Every glob resolves inside the + worktree" is **vacuously true** over the empty set, so a two-valued P1 returns PASS for a + plan that declares no capability at all. That is UNDETERMINED here. + """ + if not scope: + return Term( + "P1", + Result.UNDETERMINED, + "write_scope is absent or empty — no capability is declared, so " + "'every glob resolves inside the worktree' is vacuously true and answers nothing", + ) + escaping = [] + for entry in scope: + glob, _ = deglue(entry) + if escapes_root(glob): + escaping.append(entry) + if escaping: + return Term( + "P1", + Result.FAIL, + "write_scope entries do not resolve inside the worktree: " + + ", ".join(repr(e) for e in escaping), + ) + return Term("P1", Result.PASS, f"{len(scope)} globs, all repo-relative and root-confined") + + +def check_p2(scope: list[str]) -> Term: + """P2 — record/enforcement-free: `scope ∩ FORBIDDEN_SCOPE = ∅`. + + Checked in **both** glob directions, both through `_lib.matches_any`, because neither + alone is sufficient: (a) the entry read as a path against the forbidden patterns catches + `eval/foo.py`; (b) a forbidden witness against the entry read as a pattern catches a + glob that *covers* a forbidden surface without naming it (`.claude/**`, `docs/**`, `**`). + + ⚑ Degenerate inputs. An empty `write_scope` makes the intersection vacuously empty → + UNDETERMINED. An entry with a glued inline comment (`finding-0085`) matches nothing, so + the intersection is *also* vacuously empty even on a scope that names `eval/` → FAIL, + because the declared capability is not what it reads as. + """ + if not scope: + return Term( + "P2", + Result.UNDETERMINED, + "write_scope is absent or empty — 'scope ∩ forbidden = ∅' is vacuously true " + "over an empty scope and answers nothing", + ) + hits: list[str] = [] + for entry in scope: + glob, glued = deglue(entry) + if glued: + hits.append( + f"{entry!r} carries an inline comment glued to the glob (finding-0085 / " + "bp-066): it matches nothing, so the forbidden-set intersection would be " + "vacuously empty and the declared scope is not what it reads as" + ) + if not glob: + continue + if matches_any(glob, list(FORBIDDEN_SCOPE)): + hits.append(f"{glob!r} lies inside the §2.4 forbidden set") + continue + covered = [w for w in FORBIDDEN_WITNESSES if matches_any(w, [glob])] + if covered: + hits.append(f"{glob!r} covers forbidden surface(s) {', '.join(covered)}") + if hits: + return Term("P2", Result.FAIL, "; ".join(hits)) + return Term("P2", Result.PASS, "scope is disjoint from the §2.4 record/enforcement set") + + +def section_seven(text: str) -> str | None: + """The plan's §7 body, or None if the plan has no §7 heading.""" + match = _SECTION_7.search(text) + if match is None: + return None + rest = text[match.start() :] + nxt = _ANY_SECTION.search(rest, 1) + return rest if nxt is None else rest[: nxt.start()] + + +def item_bodies(section: str) -> list[str]: + """The `### Item ` blocks of a §7 body, in order.""" + starts = [m.start() for m in _ITEM_HEADING.finditer(section)] + bounds = starts + [len(section)] + return [section[bounds[i] : bounds[i + 1]] for i in range(len(starts))] + + +def item_label(body: str) -> str: + """The item's heading line, trimmed, for use in a diagnostic.""" + head = body.splitlines()[0] if body.splitlines() else "### Item ?" + return head.lstrip("# ").strip() + + +def normalize_flag(value: str) -> str: + """P3's normalization, pinned (plan §6). + + Truncate at a run-on bolded field, then strip surrounding emphasis and a trailing period, + then lowercase. `no`, `No`, `No.`, `**No.**` and `No. **Parallelizable?** yes` all become + `no`; `No (reads the corpus)` and `Reads only.` do not, and must not — reading the first + two characters of an English sentence is not a blast-radius decision. + """ + v = value.strip() + for marker in _TRUNCATE_AT: + idx = v.find(marker) + if idx != -1: + v = v[:idx] + return v.strip().strip("*").strip().rstrip(".").strip().lower() + + +def check_p3(section: str | None) -> Term: + """P3 — no stored-data blast, as corrected by `finding-0263` (see the module banner). + + ⚑ Degenerate inputs. A plan with zero `### Item ` headings makes "every item carries the + flag as no" **vacuously true** → UNDETERMINED. And the literal §2.4 reading — grep for + `touches_stored_data:`, find nothing, see no `true`, return PASS — passes on every plan + in the repository; it is not implemented here, and a test constructs a plan whose §7 says + `**Touches stored data?** Yes — rewrites the vector store` and requires FAIL. + """ + if section is None or not section.strip(): + return Term( + "P3", + Result.UNDETERMINED, + "the plan has no §7 section, so 'every item carries the flag' is vacuous", + ) + bodies = item_bodies(section) + if not bodies: + return Term( + "P3", + Result.UNDETERMINED, + "§7 carries zero '### Item ' headings — 'every item carries the flag as no' is " + "vacuously true over the empty set and answers nothing", + ) + fails: list[str] = [] + undetermined: list[str] = [] + for body in bodies: + label = item_label(body) + values = [m.group("value") for m in _FLAG_LINE.finditer(body)] + if len(values) == 0: + undetermined.append(f"{label}: no '**Touches stored data?**' line") + continue + if len(values) > 1: + undetermined.append(f"{label}: {len(values)} '**Touches stored data?**' lines") + continue + normalized = normalize_flag(values[0]) + if normalized != "no": + fails.append(f"{label}: {values[0].strip()!r} normalizes to {normalized!r}, not 'no'") + if fails: + return Term("P3", Result.FAIL, "; ".join(fails)) + if undetermined: + return Term("P3", Result.UNDETERMINED, "; ".join(undetermined)) + return Term("P3", Result.PASS, f"all {len(bodies)} items carry the flag as exactly 'no'") + + +def check_p4(section: str | None) -> Term: + """P4 — no live-state mutation, as a lexical scan that admits what it cannot see. + + §2.4's P4 names "no acceptance step or action runs `deploy`, `palace` lifecycle mutation, + or any credentialed external call". There is no structured acceptance field on a plan and + this plan does not invent one (§9 non-goal 2), so this is a token scan over §7's prose + with `LIVE_STATE_TOKENS` — **fenced blocks included**, because a command in a fence is + still a command and a fence is exactly where an acceptance step lives. + + ⚑ Degenerate inputs. An absent or empty §7 makes "no acceptance step runs `deploy`" + vacuously true → UNDETERMINED. And a scanner that skips fences — the natural "ignore + code" instinct — passes a plan whose acceptance is literally ```mind-palace deploy```; + fences are deliberately NOT skipped, and a test pins that. + """ + if section is None or not section.strip(): + return Term( + "P4", + Result.UNDETERMINED, + "the plan has no §7 section, so 'no acceptance step runs deploy' is vacuous", + ) + if not item_bodies(section): + return Term( + "P4", + Result.UNDETERMINED, + "§7 carries zero '### Item ' headings — there is no acceptance prose to scan, so " + "'no acceptance step runs deploy' is vacuously true and answers nothing", + ) + hits: list[str] = [] + for label, pattern in LIVE_STATE_TOKENS: + match = re.search(pattern, section, re.I) + if match is not None: + hits.append(f"{label} ({match.group(0).strip()!r})") + if hits: + return Term( + "P4", + Result.FAIL, + "§7 prose contains live-state token(s): " + ", ".join(hits), + ) + return Term("P4", Result.PASS, "no live-state token in §7 (lexical scan, fences included)") + + +def check_p5(front_matter: dict[str, object]) -> Term: + """P5 — bounded: `session_budget` parses as an integer >= 1. + + A trailing YAML `#` comment is tolerated because `_lib._scalar` leaves one glued to an + unquoted scalar by design; rejecting legal YAML would enforce a spelling rather than a + property. `"null"`, empty and absent are the same thing: UNDETERMINED. + + ⚑ Not checked, and said so rather than implied: un-self-extendability is a property of + the runtime, not of the plan text (plan §11 row 4). + """ + raw = front_matter.get("session_budget") + if _is_absent(raw): + return Term( + "P5", + Result.UNDETERMINED, + "session_budget is absent, empty or null — the run's bound is undeclared", + ) + match = re.match(r"^(-?\d+)[ \t]*(?:#.*)?$", str(raw).strip()) + if match is None: + return Term("P5", Result.FAIL, f"session_budget {str(raw).strip()!r} is not an integer") + budget = int(match.group(1)) + if budget < 1: + return Term("P5", Result.FAIL, f"session_budget is {budget}, not a finite bound >= 1") + return Term( + "P5", + Result.PASS, + f"session_budget is {budget} (the plan's declared bound; the runtime half is unchecked)", + ) + + +def conjoin(results: dict[str, Result]) -> Result: + """The conjunction, pinned (plan §6): PASS **iff** the term set is exactly `TERM_NAMES` + and every term is PASS. Everything else — including any `UNDETERMINED` — is FAIL. + + ⚑ The signature is the guard. A `Sequence[Result]` overload would make `all(...)` return + True over the empty sequence, so a caller that computed nothing would receive PASS; and + the natural `all(t is not Result.FAIL)` returns PASS when every term is UNDETERMINED, + i.e. when nothing was determined about anything. Requiring the complete keyed term set + makes both unreachable rather than merely untested. There is no other conjunction in this + module and no code path returns overall PASS while any term is not PASS. + """ + if set(results) != set(TERM_NAMES): + return Result.FAIL + return Result.PASS if all(results[n] is Result.PASS for n in TERM_NAMES) else Result.FAIL + + +@dataclass +class Evaluation: + """All five terms for one plan, plus the scope they were computed over.""" + + terms: dict[str, Term] = field(default_factory=dict) + scope: list[str] = field(default_factory=list) + + @property + def overall(self) -> Result: + return conjoin({name: term.result for name, term in self.terms.items()}) + + +def evaluate(text: str) -> Evaluation: + """Evaluate P1–P5 over one plan's text. Read-only; nothing here writes a file.""" + front_matter = parse_front_matter(text) + scope = write_scope(front_matter) + section = section_seven(text) + terms = { + "P1": check_p1(scope), + "P2": check_p2(scope), + "P3": check_p3(section), + "P4": check_p4(section), + "P5": check_p5(front_matter), + } + return Evaluation(terms=terms, scope=scope) + + +def report_line(evaluation: Evaluation) -> str: + """The single capsule-ready `achievable:` line (plan §6). + + One line, because seven other fields share the capsule's 40-line / 300-word budget + (`scripts/capsule.py:62-63`). ⚑ Those caps bound **shape, not bytes** (`finding-0219` / + `oq-0054`, open with no recorded ruling), so this assumes no character bound exists and + simply keeps its own output short. + + The marks are rendered from the computed terms, never from a template, so the line for a + FAIL plan differs from the line for a PASS plan in exactly the term positions. The + caveats are standing — they hold for every plan — so §6's `"no caveats"` alternative is + never taken; that is stated here rather than left as a surprising absence. + """ + marks = " ".join( + f"{name} {evaluation.terms[name].result.value}" + ("(lexical)" if name == "P4" else "") + for name in TERM_NAMES + ) + return f"achievable: {len(evaluation.scope)} globs; {marks}; {REPORT_CAVEATS}" + + +def diagnostics(evaluation: Evaluation) -> list[str]: + """One line per non-PASS term, in term order: predicate, result, evidence.""" + return [ + f"{name} {evaluation.terms[name].result.value}: {evaluation.terms[name].evidence}" + for name in TERM_NAMES + if evaluation.terms[name].result is not Result.PASS + ] + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser( + prog="autopilot_eligibility.py", + description="§2.4's P1-P5 autopilot-eligibility predicate over a build plan.", + ) + sub = parser.add_subparsers(dest="cmd", required=True) + for name, help_text in ( + ("check", "exit 0 iff all five predicates PASS; else one diagnostic line per term"), + ("report", "print the capsule-ready one-line `achievable:` block"), + ): + cmd = sub.add_parser(name, help=help_text) + cmd.add_argument("plan", help="path to a build plan's plan.md") + args = parser.parse_args(argv) + + try: + text = Path(args.plan).read_text(encoding="utf-8") + except OSError as exc: + print(f"error: cannot read {args.plan}: {exc}", file=sys.stderr) + return 1 + + evaluation = evaluate(text) + if args.cmd == "report": + print(report_line(evaluation)) + for line in P3_CORRECTION_BANNER.splitlines(): + print(f"# {line}" if line.strip() else "#", file=sys.stderr) + else: + for line in diagnostics(evaluation): + print(line) + return 0 if evaluation.overall is Result.PASS else 1 + + +if __name__ == "__main__": # pragma: no cover + raise SystemExit(main()) diff --git a/tests/unit/test_autopilot_eligibility.py b/tests/unit/test_autopilot_eligibility.py new file mode 100644 index 0000000..b811caf --- /dev/null +++ b/tests/unit/test_autopilot_eligibility.py @@ -0,0 +1,796 @@ +"""bp-137 — the autopilot-eligibility predicate: falsifiers, degenerate inputs, and the census. + +Every item of bp-137 delivers a **gate**, so the false-success rule applies to all four +(`docs/brainstorms/the-false-success-rule.md:17-31`): each check's degenerate input — the case +on which it would pass *without testing its claim* — is named and asserted to redden. The +degenerate inputs here are all the same shape: a term whose evidence is absent rather than +violated, over which "for every x, P(x)" is vacuously true. +""" + +from __future__ import annotations + +import ast +import itertools +import subprocess +import sys +from collections.abc import Sequence +from pathlib import Path + +import pytest + +REPO = Path(__file__).resolve().parents[2] +sys.path.insert(0, str(REPO / "scripts")) +sys.path.insert(0, str(REPO / ".claude" / "hooks")) + +import _lib # type: ignore[import-not-found] # noqa: E402 +from autopilot_eligibility import ( # type: ignore[import-not-found] # noqa: E402 + FORBIDDEN_SCOPE, + FORBIDDEN_WITNESSES, + LIVE_STATE_TOKENS, + P3_CORRECTION_BANNER, + TERM_NAMES, + Evaluation, + Result, + Term, + check_p1, + check_p2, + check_p3, + check_p4, + check_p5, + conjoin, + deglue, + diagnostics, + escapes_root, + evaluate, + item_bodies, + main, + matches_any, + normalize_flag, + report_line, + section_seven, + write_scope, +) + +TOOL = REPO / "scripts" / "autopilot_eligibility.py" +PLANS = sorted((REPO / "docs" / "build-plans").glob("*/plan.md")) + + +# ------------------------------------------------------------------------------------------ +# Fixture construction — a minimal but real-shaped plan +# ------------------------------------------------------------------------------------------ + + +def make_plan( + *, + scope: Sequence[str] | None = ("scripts/foo.py",), + budget: str | None = "1", + items: Sequence[str | None] | None = None, + extra_item_prose: str = "", + omit_section_7: bool = False, + zero_items: bool = False, +) -> str: + """Build a plan's text. `items` is one entry per §7 item: the raw text after + `**Touches stored data?**`, or None for an item that carries no flag line at all. + `scope=None` omits the `write_scope` key entirely; `scope=[]` writes `write_scope: []`.""" + lines = ["---", "type: build-plan", "id: bp-999", "status: ready"] + if scope is not None: + if len(scope) == 0: + lines.append("write_scope: []") + else: + lines.append("write_scope:") + lines += [f" - {entry}" for entry in scope] + if budget is not None: + lines.append(f"session_budget: {budget}") + lines += ["---", "", "# Build Plan — fixture", "", "## 1. Objective", "", "A fixture.", ""] + if not omit_section_7: + lines += ["## 7. Items", ""] + if not zero_items: + for i, flag in enumerate(items if items is not None else ["No."], start=1): + lines += [ + f"### Item {i} — fixture item", + "", + "- **Objective:** one line", + "- **Acceptance test:** `uv run pytest -q` green", + ] + if flag is not None: + lines.append(f"- **Touches stored data?** {flag}") + if extra_item_prose: + lines.append(extra_item_prose) + lines.append("") + lines += ["## 9. Non-goals", "", "1. Nothing.", ""] + return "\n".join(lines) + "\n" + + +CLEAN_ITEM = "No." + + +# ------------------------------------------------------------------------------------------ +# Item 14 — P1: repo-confined +# ------------------------------------------------------------------------------------------ + + +@pytest.mark.parametrize( + "entry", + [ + "/etc/passwd", + "/Users/someone/elsewhere/**", + "../outside/**", + "docs/../../outside.py", + "~/secrets.txt", + "C:/windows/**", + ], +) +def test_p1_fails_for_a_glob_that_leaves_the_worktree(entry: str) -> None: + term = check_p1([entry]) + assert term.result is Result.FAIL, term + assert entry in term.evidence + + +@pytest.mark.parametrize( + "entry", + [ + "scripts/autopilot_eligibility.py", + "tests/unit/**", + "**/conftest.py", + "docs/build-plans/bp-137/**", + "core/stores/*.py", + "docs/..", + ], +) +def test_p1_passes_for_ordinary_repo_relative_globs(entry: str) -> None: + assert check_p1([entry]).result is Result.PASS + + +def test_p1_degenerate_empty_scope_is_undetermined_not_pass() -> None: + """⚑ THE DEGENERATE INPUT (plan §7 Item 14). "Every glob resolves inside the worktree" is + VACUOUSLY TRUE over an empty scope, so a two-valued P1 returns PASS for a plan that + declares no capability at all. Absence must refuse exactly as violation does.""" + for scope in ([], list(write_scope({})), write_scope({"write_scope": "null"})): + term = check_p1(list(scope)) + assert term.result is Result.UNDETERMINED, term + assert term.result is not Result.PASS + assert "vacuous" in term.evidence + + +def test_p1_reads_absent_empty_and_null_write_scope_all_as_absent() -> None: + assert write_scope({}) == [] + assert write_scope({"write_scope": ""}) == [] + assert write_scope({"write_scope": "null"}) == [] + assert write_scope({"write_scope": []}) == [] + assert write_scope({"write_scope": ["a.py", " ", "b.py"]}) == ["a.py", "b.py"] + + +def test_escapes_root_is_pure_string_arithmetic() -> None: + assert escapes_root("") is True + assert escapes_root(" ") is True + assert escapes_root("a/../b") is False + assert escapes_root("a/../../b") is True + assert escapes_root("a\\..\\..\\b") is True # backslashes normalized like glob_match does + + +# ------------------------------------------------------------------------------------------ +# Item 14 — P2: record/enforcement-free +# ------------------------------------------------------------------------------------------ + + +@pytest.mark.parametrize("member", list(FORBIDDEN_SCOPE)) +def test_p2_fails_for_each_forbidden_member_named_directly(member: str) -> None: + term = check_p2([member]) + assert term.result is Result.FAIL, term + + +@pytest.mark.parametrize( + "member_path", + [ + "CLAUDE.md", + ".claude/hooks/_lib.py", + ".claude/settings.json", + "docs/design-notes/agent-workflow.md", + "eval/metrics.py", + "eval/some/deep/new_file.py", + ], +) +def test_p2_fails_for_a_concrete_path_inside_a_forbidden_tree(member_path: str) -> None: + """Direction (a): the entry read as a PATH against the forbidden patterns. `eval/…` deep + files are matched by no witness, so only this direction sees them.""" + assert check_p2([member_path]).result is Result.FAIL + + +@pytest.mark.parametrize("covering", [".claude/**", "docs/**", "**", ".claude/hooks/*.py"]) +def test_p2_fails_for_a_glob_that_covers_a_forbidden_surface_without_naming_it( + covering: str, +) -> None: + """Direction (b): a forbidden WITNESS against the entry read as a pattern. `.claude/**` + is not matched by the pattern `.claude/hooks/**`, so direction (a) alone would miss it — + which is why P2 checks both, and why this test would redden if either were dropped.""" + term = check_p2([covering]) + assert term.result is Result.FAIL, term + assert "covers forbidden surface" in term.evidence or "forbidden set" in term.evidence + + +def test_p2_passes_for_a_disjoint_scope() -> None: + term = check_p2(["scripts/autopilot_eligibility.py", "tests/unit/**", "core/stores/*.py"]) + assert term.result is Result.PASS, term + + +def test_p2_degenerate_empty_scope_is_undetermined_not_pass() -> None: + """⚑ THE DEGENERATE INPUT. "scope ∩ forbidden = ∅" is VACUOUSLY TRUE over an empty scope.""" + term = check_p2([]) + assert term.result is Result.UNDETERMINED, term + assert term.result is not Result.PASS + assert "vacuously true" in term.evidence + + +def test_p2_degenerate_glued_inline_comment_fails_and_names_finding_0085() -> None: + """⚑ THE SECOND DEGENERATE INPUT (finding-0085 / bp-066). `- eval/metrics.py # absorbed` + reaches the guard with the comment GLUED to the glob, so it matches nothing — the + forbidden-set intersection is empty and a naive P2 passes a scope that names `eval/`.""" + term = check_p2(["eval/metrics.py # absorbed into the sweep"]) + assert term.result is Result.FAIL, term + assert "finding-0085" in term.evidence + # And the vacuity is real: the glued entry genuinely matches nothing under the guard. + assert not matches_any("eval/metrics.py", ["eval/metrics.py # absorbed into the sweep"]) + + +def test_p2_glued_comment_fails_even_on_an_otherwise_harmless_glob() -> None: + """A glued entry means the declared capability is not what it reads as, whatever it + names. Refusing on the ambiguity is invariant 7 at the gate's mouth.""" + term = check_p2(["scripts/foo.py # just a note"]) + assert term.result is Result.FAIL, term + assert "finding-0085" in term.evidence + + +def test_deglue_splits_the_footgun_and_leaves_a_clean_entry_alone() -> None: + assert deglue("eval/metrics.py # absorbed") == ("eval/metrics.py", True) + assert deglue(" scripts/foo.py ") == ("scripts/foo.py", False) + assert deglue("# only a comment") == ("", True) + + +# ------------------------------------------------------------------------------------------ +# Item 14 — the matcher is the guard's own, not a re-implementation +# ------------------------------------------------------------------------------------------ + + +def test_the_matcher_is_lib_matches_any_by_identity() -> None: + """Two matchers that disagree mean the predicate blesses a scope `scope-guard` reads + differently — a security-relevant duplication (plan §2 DRY audit).""" + import autopilot_eligibility + + assert autopilot_eligibility.matches_any is _lib.matches_any + + +def test_the_module_defines_no_local_glob_implementation() -> None: + """The AST half of the same assertion: no local glob function, no `fnmatch`, no `glob`.""" + tree = ast.parse(TOOL.read_text(encoding="utf-8")) + defined = { + node.name + for node in ast.walk(tree) + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) + } + for forbidden in ("glob_match", "_seg_match", "fnmatch", "_glob", "match_glob"): + assert forbidden not in defined, f"{forbidden} re-derives the guard's glob semantics" + imported = _imported_top_level(tree) + assert "fnmatch" not in imported and "glob" not in imported + + +PROBE_PATHS = [ + "CLAUDE.md", + ".claude/hooks/_lib.py", + ".claude/settings.json", + "docs/design-notes/agent-workflow.md", + "docs/build-plans/bp-137/plan.md", + "eval/golden.py", + "eval/metrics.py", + "core/stores/sourceset.py", + "scripts/board.py", + "tests/unit/test_capsule.py", + "a/deep/nested/path/file.py", +] + + +def test_p1_p2_agree_with_the_guards_matcher_on_five_real_write_scopes() -> None: + """⚑ ITEM 14's FALSIFIER, drilled. Take five real `write_scope` lists from the tree, run + `_lib.matches_any` and this predicate's own covering decision over a fixed probe set, and + assert identical verdicts. A disagreement is a change to the guard's semantics and is not + this plan's to make (plan §10) — it would stop the build, not be tuned away.""" + real: list[list[str]] = [] + for plan in PLANS: + scope = write_scope(_lib.parse_front_matter(plan.read_text(encoding="utf-8"))) + if scope: + real.append(scope) + if len(real) == 5: + break + assert len(real) == 5, "expected at least five real plans with a non-empty write_scope" + + import autopilot_eligibility + + for scope in real: + for probe in PROBE_PATHS: + guard = _lib.matches_any(probe, scope) + ours = autopilot_eligibility.matches_any(probe, scope) + assert guard == ours, f"matcher disagreement on {probe!r} vs {scope!r}" + + # The semantic bridge, not just the identity: if the GUARD would let this scope write + # a forbidden surface, P2 must refuse the plan. A P2 that called such a scope clean is + # exactly the falsifier — the predicate measuring its own glob code, not the guard's + # capability. + guard_reaches_forbidden = any( + _lib.matches_any(witness, scope) for witness in FORBIDDEN_WITNESSES + ) + if guard_reaches_forbidden: + assert check_p2(list(scope)).result is Result.FAIL, scope + + +def test_item_bodies_splits_section_7_at_each_item_heading() -> None: + section = section_seven(make_plan(items=["No.", "No.", "No."])) + assert section is not None + bodies = item_bodies(section) + assert len(bodies) == 3 + assert all(body.startswith("### Item ") for body in bodies) + assert section_seven(make_plan(omit_section_7=True)) is None + + +# ------------------------------------------------------------------------------------------ +# Item 15 — P3: the stored-data flag against the pinned regex (finding-0263) +# ------------------------------------------------------------------------------------------ + + +@pytest.mark.parametrize( + "value", + [ + "no", + "No", + "No.", + "**No.**", + "No. **Parallelizable?** yes **Depends on:** none", + "no **Depends on:** none", + "**No.** **Parallelizable?** no", + ], +) +def test_p3_passes_only_on_a_value_that_normalizes_to_exactly_no(value: str) -> None: + assert normalize_flag(value) == "no" + assert check_p3(section_seven(make_plan(items=[value]))).result is Result.PASS + + +@pytest.mark.parametrize( + "value", + [ + "yes", + "Yes — a new SQLite table under data/", + "Reads only.", + "No (reads the corpus)", + "No — reads the vault", + "No, but it reads the store", + "Not directly.", + ], +) +def test_p3_fails_on_every_hedged_spelling_from_the_census(value: str) -> None: + term = check_p3(section_seven(make_plan(items=[value]))) + assert term.result is Result.FAIL, term + assert term.result is not Result.PASS + + +def test_p3_undetermined_for_an_item_with_no_flag_line() -> None: + term = check_p3(section_seven(make_plan(items=[CLEAN_ITEM, None]))) + assert term.result is Result.UNDETERMINED, term + assert "no '**Touches stored data?**' line" in term.evidence + + +def test_p3_undetermined_for_two_flag_lines_in_one_item() -> None: + plan = make_plan(items=["No."], extra_item_prose="- **Touches stored data?** No.") + term = check_p3(section_seven(plan)) + assert term.result is Result.UNDETERMINED, term + assert "2 '**Touches stored data?**' lines" in term.evidence + + +def test_p3_degenerate_zero_items_is_undetermined_not_pass() -> None: + """⚑ THE DEGENERATE INPUT (plan §7 Item 15). "Every item carries the flag as `no`" is + VACUOUSLY TRUE over a plan with no items, so a naive P3 returns PASS for a plan with no + §7 at all. Absence refuses exactly as violation does.""" + for plan in (make_plan(zero_items=True), make_plan(omit_section_7=True)): + term = check_p3(section_seven(plan)) + assert term.result is Result.UNDETERMINED, term + assert term.result is not Result.PASS + + +def test_p3_degenerate_the_literal_2_4_reading_would_pass_the_vector_store_plan() -> None: + """⚑ THE SECOND DEGENERATE INPUT. §2.4's literal check — grep for `touches_stored_data:`, + find nothing, see no `true`, return PASS — passes on EVERY plan in the repository. This + test is the one that reddens if the implementation ever regresses to the front-matter key: + it asserts FAIL on a plan whose §7 says it rewrites the vector store, and demonstrates in + the same breath that the literal reading finds nothing to object to.""" + plan = make_plan(items=["Yes — rewrites the vector store"]) + assert check_p3(section_seven(plan)).result is Result.FAIL + assert "touches_stored_data:" not in plan # the literal field simply is not there + assert "true" not in plan.lower().split("## 7.")[1] # ...and there is no `true` to catch + + +def test_p3_fail_dominates_undetermined_within_one_plan() -> None: + term = check_p3(section_seven(make_plan(items=["Yes", None]))) + assert term.result is Result.FAIL + + +def test_normalize_flag_does_not_prefix_match() -> None: + """m1 of the mutation campaign: `value.startswith("no")` in place of exact equality is + caught here and by the `No (reads the corpus)` fixture above.""" + assert normalize_flag("No (reads the corpus)") != "no" + assert normalize_flag("Not directly.") != "no" + assert normalize_flag("No — reads the vault") != "no" + + +def test_normalize_flag_truncation_at_a_run_on_bolded_field_is_load_bearing() -> None: + """m3: dropping the truncation makes the 57-plan run-on form fail. Pin it explicitly.""" + run_on = "No. **Parallelizable?** yes **Depends on:** none" + assert normalize_flag(run_on) == "no" + assert run_on.strip().strip("*").strip().rstrip(".").strip().lower() != "no" + + +def test_module_docstring_carries_the_finding_0263_correction_banner_verbatim() -> None: + """Plan §4: the ratified note is agent-immutable (A8), so the correction is carried in the + tool's own docstring and repeated in its report output. The banner is SLICED from + `__doc__`, so docstring and report cannot drift.""" + import autopilot_eligibility + + doc = autopilot_eligibility.__doc__ or "" + assert P3_CORRECTION_BANNER + assert P3_CORRECTION_BANNER in doc + for phrase in ( + "DOES NOT EXIST", + "appears in ZERO", + "exactly `no`", + "AUTHORITATIVE", + "`spec-defect`, never a silent re-interpretation", + "LEXICAL scan", + "Un-self-extendability", + ): + assert phrase in P3_CORRECTION_BANNER, phrase + + +# ------------------------------------------------------------------------------------------ +# Item 16 — P4: the lexical scan, and P5: the bound +# ------------------------------------------------------------------------------------------ + + +@pytest.mark.parametrize( + "prose", + [ + "- run `mind-palace deploy` at the end", + "- DEPLOY the stack", + "- `palace restart` after the change", + "- palace stop, then start again", + "- read the token with `security find-generic-password -s x`", + "- call `get_secret('x')`", + "- upload with boto3", + "- run `aws s3 cp ...`", + "- fetch it with curl https://example.com", + "- shell out via subprocess.run", + ], +) +def test_p4_fails_on_every_deny_list_token(prose: str) -> None: + term = check_p4(section_seven(make_plan(items=[CLEAN_ITEM], extra_item_prose=prose))) + assert term.result is Result.FAIL, term + + +def test_p4_passes_for_a_plan_whose_section_7_contains_no_token() -> None: + term = check_p4(section_seven(make_plan(items=[CLEAN_ITEM]))) + assert term.result is Result.PASS, term + + +def test_p4_degenerate_absent_or_empty_section_7_is_undetermined_not_pass() -> None: + """⚑ THE DEGENERATE INPUT (plan §7 Item 16). "No acceptance step runs `deploy`" is + VACUOUSLY TRUE when there is nothing to scan.""" + for plan in (make_plan(omit_section_7=True), make_plan(zero_items=True)): + term = check_p4(section_seven(plan)) + assert term.result is Result.UNDETERMINED, term + assert term.result is not Result.PASS + + +def test_p4_degenerate_a_token_inside_a_fenced_block_still_fails() -> None: + """⚑ THE SECOND DEGENERATE INPUT. A scanner that skips fences — the natural "ignore code" + instinct — passes a plan whose acceptance is literally a fenced `mind-palace deploy`. The + second assertion is what would redden if fence-skipping were introduced: the fence is the + ONLY carrier of the token here, so a fence-blind scan sees a clean §7.""" + fenced = "\n```\nmind-palace deploy\n```\n" + section = section_seven(make_plan(items=[CLEAN_ITEM], extra_item_prose=fenced)) + assert section is not None + assert check_p4(section).result is Result.FAIL + + defenced = _strip_fences(section) + assert "deploy" not in defenced.lower(), "fixture no longer isolates the token to the fence" + assert check_p4(defenced).result is Result.PASS + + +def _strip_fences(text: str) -> str: + """A fence-skipping scanner's view of the text — used only to prove what one would miss.""" + out, inside = [], False + for line in text.splitlines(): + if line.lstrip().startswith("```"): + inside = not inside + continue + if not inside: + out.append(line) + return "\n".join(out) + + +def test_p4_token_table_is_the_pinned_deny_list() -> None: + labels = {label for label, _ in LIVE_STATE_TOKENS} + assert labels == { + "deploy", + "palace lifecycle mutation", + "keychain read", + "secret read", + "aws sdk", + "aws cli", + "network call", + "shell-out", + } + + +def test_p5_passes_for_a_finite_bound() -> None: + assert check_p5({"session_budget": "1"}).result is Result.PASS + assert check_p5({"session_budget": "3 # comment"}).result is Result.PASS + + +@pytest.mark.parametrize("budget", ["0", "-1", "1.5", "one", "many", "1 2"]) +def test_p5_fails_for_a_non_positive_or_non_integer_bound(budget: str) -> None: + assert check_p5({"session_budget": budget}).result is Result.FAIL + + +@pytest.mark.parametrize("budget", [None, "", " ", "null", "none", "~"]) +def test_p5_degenerate_absent_empty_or_null_is_undetermined_not_pass(budget: str | None) -> None: + """⚑ `_lib.parse_front_matter` returns a YAML `null` as the literal string `"null"` + (§3 Q4), so an unfilled budget field looks like a value. Undetermined ⇒ refuse.""" + front_matter: dict[str, object] = {} if budget is None else {"session_budget": budget} + term = check_p5(front_matter) + assert term.result is Result.UNDETERMINED, term + assert term.result is not Result.PASS + + +def test_p5_evidence_states_that_the_runtime_half_is_unchecked() -> None: + assert "runtime half is unchecked" in check_p5({"session_budget": "1"}).evidence + + +# ------------------------------------------------------------------------------------------ +# Item 17 — the conjunction +# ------------------------------------------------------------------------------------------ + + +def test_conjunction_over_all_243_term_combinations() -> None: + """⚑ THE LOAD-BEARING TEST. All 3**5 = 243 combinations: exactly ONE is overall PASS, and + no combination containing an UNDETERMINED is. This is the assertion that the natural + `all(t is not Result.FAIL)` — which returns PASS when every term is UNDETERMINED, i.e. + when nothing was determined about anything — is impossible here.""" + passes = [] + for combo in itertools.product(Result, repeat=len(TERM_NAMES)): + results = dict(zip(TERM_NAMES, combo, strict=True)) + overall = conjoin(results) + assert overall in (Result.PASS, Result.FAIL) + if overall is Result.PASS: + passes.append(results) + if any(r is Result.UNDETERMINED for r in combo): + assert overall is Result.FAIL, results + assert len(passes) == 1 + assert passes[0] == dict.fromkeys(TERM_NAMES, Result.PASS) + + +def test_the_naive_conjunction_would_admit_thirty_two_combinations() -> None: + """The mutation, stated as arithmetic: `all(t is not FAIL)` admits 2**5 = 32 combinations + including the all-UNDETERMINED one. `conjoin` admits exactly 1. If the two counts ever + agree, the conjunction has become two-valued in disguise.""" + naive = [ + combo + for combo in itertools.product(Result, repeat=len(TERM_NAMES)) + if all(r is not Result.FAIL for r in combo) + ] + assert len(naive) == 32 + assert conjoin(dict.fromkeys(TERM_NAMES, Result.UNDETERMINED)) is Result.FAIL + + +@pytest.mark.parametrize( + "results", + [ + {}, + {"P1": Result.PASS}, + dict.fromkeys(("P1", "P2", "P3", "P4"), Result.PASS), + dict.fromkeys(("P1", "P2", "P3", "P4", "P5", "P6"), Result.PASS), + dict.fromkeys(("P1", "P2", "P3", "P4", "PX"), Result.PASS), + ], +) +def test_conjunction_refuses_an_incomplete_or_wrong_term_set(results: dict[str, Result]) -> None: + """⚑ Absence at the conjunction itself. `all(...)` over an empty sequence is True — the + purest vacuous pass in the module — so `conjoin` requires the COMPLETE keyed term set as a + precondition rather than trusting its caller to have computed one.""" + assert conjoin(results) is Result.FAIL + + +# ------------------------------------------------------------------------------------------ +# Item 17 — the report +# ------------------------------------------------------------------------------------------ + + +def _clean_plan(n_globs: int = 1) -> str: + return make_plan( + scope=[f"scripts/generated_{i}.py" for i in range(n_globs)], + items=[CLEAN_ITEM, CLEAN_ITEM], + ) + + +def test_a_fully_clean_plan_passes_all_five_terms() -> None: + evaluation = evaluate(_clean_plan()) + assert evaluation.overall is Result.PASS, diagnostics(evaluation) + assert all(t.result is Result.PASS for t in evaluation.terms.values()) + + +def test_report_is_one_line_under_200_chars_for_a_plan_with_twenty_globs() -> None: + evaluation = evaluate(_clean_plan(20)) + line = report_line(evaluation) + assert "\n" not in line + assert len(line) < 200, len(line) + assert line.startswith("achievable: 20 globs;") + + +def test_report_line_is_rendered_from_the_computed_terms_not_a_template() -> None: + """⚑ THE SECOND DEGENERATE INPUT for Item 17: a `report` that prints `P1 pass P2 pass …` + from a template rather than from the computed terms. The FAIL plan's line must differ from + the PASS plan's in exactly the term positions that differ.""" + good = report_line(evaluate(_clean_plan())) + bad = report_line( + evaluate( + make_plan(scope=["eval/**"], items=["Yes — rewrites the vector store"], budget="0") + ) + ) + assert good != bad + assert "P1 pass P2 pass P3 pass P4 pass(lexical) P5 pass" in good + assert "P2 fail" in bad and "P3 fail" in bad and "P5 fail" in bad + + +def test_report_line_carries_the_standing_caveats() -> None: + line = report_line(evaluate(_clean_plan())) + assert "P3=finding-0263 form" in line + assert "P4 lexical" in line + assert "P5 plan-side bound only" in line + + +def test_report_line_worst_case_width_is_bounded() -> None: + evaluation = Evaluation( + terms={n: Term(n, Result.UNDETERMINED, "") for n in TERM_NAMES}, + scope=[f"g{i}/**" for i in range(20)], + ) + assert len(report_line(evaluation)) < 200 + + +def test_diagnostics_name_every_non_pass_term_in_order() -> None: + evaluation = evaluate(make_plan(scope=[], items=None, budget=None, zero_items=True)) + lines = diagnostics(evaluation) + assert [line.split()[0] for line in lines] == list(TERM_NAMES) + assert all("undetermined" in line for line in lines) + + +# ------------------------------------------------------------------------------------------ +# Item 17 — the CLI +# ------------------------------------------------------------------------------------------ + + +def _run(*args: str) -> subprocess.CompletedProcess[str]: + return subprocess.run( + [sys.executable, str(TOOL), *args], capture_output=True, text=True, check=False + ) + + +def test_check_exits_zero_iff_overall_pass(tmp_path: Path) -> None: + good = tmp_path / "good.md" + good.write_text(_clean_plan(), encoding="utf-8") + bad = tmp_path / "bad.md" + bad.write_text(make_plan(scope=[], items=["Yes"]), encoding="utf-8") + + ok = _run("check", str(good)) + assert ok.returncode == 0, ok.stdout + ok.stderr + assert ok.stdout.strip() == "" + + nope = _run("check", str(bad)) + assert nope.returncode == 1 + assert "P1 undetermined:" in nope.stdout + assert "P3 fail:" in nope.stdout + + +def test_report_stdout_is_exactly_one_line_and_the_banner_goes_to_stderr(tmp_path: Path) -> None: + plan = tmp_path / "p.md" + plan.write_text(_clean_plan(), encoding="utf-8") + out = _run("report", str(plan)) + assert out.returncode == 0, out.stdout + out.stderr + assert len(out.stdout.strip().splitlines()) == 1 + assert out.stdout.startswith("achievable: ") + assert "DOES NOT EXIST" in out.stderr + assert "LEXICAL scan" in out.stderr + + +def test_main_returns_one_for_an_unreadable_plan(tmp_path: Path) -> None: + assert main(["check", str(tmp_path / "nope.md")]) == 1 + + +# ------------------------------------------------------------------------------------------ +# Item 17 — the tooling invariant (the tests/unit/test_capsule.py:414-431 precedent) +# ------------------------------------------------------------------------------------------ + + +def _imported_top_level(tree: ast.Module) -> set[str]: + imported: set[str] = set() + for node in ast.walk(tree): + if isinstance(node, ast.ImportFrom) and node.module: + imported.add(node.module.split(".")[0]) + elif isinstance(node, ast.Import): + for alias in node.names: + imported.add(alias.name.split(".")[0]) + return imported + + +def test_the_tool_imports_stdlib_and_lib_only_and_cannot_reach_a_secret() -> None: + """Absent `os` and `subprocess` there is no route to an environment variable, the + `security` CLI, or any other secret path — finding-0207's constraint held by construction + rather than by convention. §2.4: *"The agent's role in eligibility is exactly nothing."*""" + imported = _imported_top_level(ast.parse(TOOL.read_text(encoding="utf-8"))) + allowed = {"__future__", "argparse", "enum", "re", "sys", "dataclasses", "pathlib", "_lib"} + assert imported <= allowed, f"unexpected imports: {imported - allowed}" + for forbidden in ("os", "subprocess", "core", "config", "keyring", "hmac"): + assert forbidden not in imported, f"the predicate must never import {forbidden}" + + +def test_the_tool_never_writes_a_file() -> None: + """It computes a necessary condition and holds no authority: no `open(...,'w')`, no + `write_text`, no `status` flip.""" + source = TOOL.read_text(encoding="utf-8") + for forbidden in ("write_text(", "open(", "unlink(", "mkdir(", "rename("): + assert forbidden not in source, forbidden + + +def test_the_forbidden_set_is_the_verbatim_2_4_membership() -> None: + """m1 of the campaign: dropping `.claude/hooks/**` from the forbidden set is caught here + and by the covering test above. P2 is the leg that keeps autopilot out of its own cage.""" + assert set(FORBIDDEN_SCOPE) == { + "CLAUDE.md", + ".claude/hooks/**", + ".claude/settings.json", + "docs/design-notes/**", + "eval/**", + } + for witness in FORBIDDEN_WITNESSES: + assert matches_any(witness, list(FORBIDDEN_SCOPE)), witness + + +# ------------------------------------------------------------------------------------------ +# Item 17 — the census (READ-ONLY over the real tree; asserts termination, never a pass rate) +# ------------------------------------------------------------------------------------------ + + +def test_census_over_every_real_plan_terminates_with_a_result_for_each() -> None: + """Plan §7 Item 17: run the predicate over every `docs/build-plans/*/plan.md`, assert only + that it terminates and returns a `Result` for each, and print the tally. It asserts NO + pass rate: most existing plans will not pass, and that is correct — P1–P5 gates autopilot + eligibility, not repo hygiene (plan §9 non-goal 1). + + ⚑ Item 17's falsifier lives here: if EVERY real plan were UNDETERMINED on the same term, + the predicate would be measuring an authoring convention nobody follows rather than a + property — a `spec-defect` against §2.4, routed to the orchestrator, not a threshold to + relax. The assertion below is exactly that no term is constant-UNDETERMINED across the + tree, which is also the "a constant function measures nothing" keep from plan §8. + """ + assert len(PLANS) > 50, "expected the real build-plan tree" + tally: dict[str, dict[str, int]] = {n: {r.value: 0 for r in Result} for n in TERM_NAMES} + overall = {r.value: 0 for r in Result} + for plan in PLANS: + evaluation = evaluate(plan.read_text(encoding="utf-8")) + assert set(evaluation.terms) == set(TERM_NAMES), plan + for name, term in evaluation.terms.items(): + assert isinstance(term.result, Result), plan + tally[name][term.result.value] += 1 + overall[evaluation.overall.value] += 1 + + print(f"\ncensus over {len(PLANS)} plans — overall: {overall}") + for name in TERM_NAMES: + print(f" {name}: {tally[name]}") + + for name in TERM_NAMES: + assert tally[name]["undetermined"] < len(PLANS), ( + f"{name} is UNDETERMINED on every plan in the tree — the predicate is measuring an " + f"authoring convention nobody follows. STOP and file a spec-defect against §2.4." + ) From 94fffebff5464ee9f54c7380448b4ec2d4fdb62a Mon Sep 17 00:00:00 2001 From: Alberto Serrano-Calva Date: Mon, 27 Jul 2026 19:39:28 -0400 Subject: [PATCH 2/3] =?UTF-8?q?docs(bp-137):=20finding-0271=20+=20the=20bu?= =?UTF-8?q?ild=20journal=20=E2=80=94=20the=20P3=20regex's=20blind=20spot,?= =?UTF-8?q?=20measured?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Item 15's falsifier fires partially: 9 plans fold `**Touches stored data?**` onto another field's bullet, which §6's bullet-anchored regex misses, and 5 of the 25 plans P3 calls UNDETERMINED are undetermined for that reason. Not §10's "dominated by" threshold (20%), and the folded form is not the template's shape, so the build continued against §6 as pinned rather than widening the parse on a builder's authority. The miss is safe by construction and the finding says so: a folded flag leaves its item with no anchored line, so the item is UNDETERMINED and the plan FAILs. It can never produce a false PASS, not even on a folded `Yes`. Journal records the 243-combination proof, the 12/12 mutation campaign, the 137-plan census (13 pass all five — not a constant function), all four falsifier verdicts, and the evidence that the 4 remaining suite failures are the owner's OUROBOROS_HOOKS_OFF escape hatch plus a worktree-nesting artifact. --- docs/build-plans/bp-137/journal.md | 189 +++++++++++++++++++++++++++++ docs/findings/finding-0271.md | 114 +++++++++++++++++ 2 files changed, 303 insertions(+) create mode 100644 docs/findings/finding-0271.md diff --git a/docs/build-plans/bp-137/journal.md b/docs/build-plans/bp-137/journal.md index f67fe6c..ab69d9a 100644 --- a/docs/build-plans/bp-137/journal.md +++ b/docs/build-plans/bp-137/journal.md @@ -28,3 +28,192 @@ - **This predicate's output is rendered to the owner's phone.** A vacuous PASS is a false statement shown to him at the moment he decides whether to grant. Weight the degenerate-input criteria accordingly. + +--- + +## 2026-07-27 — session start (delegated builder, worktree `worktree-agent-a25e8e55678c3749c`) + +**Base:** `69a065c` (origin/main at spawn). Hooks are disabled repo-wide by owner ruling, so +write-scope discipline is held by hand and by the reviewer's pre-merge diff read. Write scope: +`scripts/autopilot_eligibility.py`, `tests/unit/test_autopilot_eligibility.py` (+ this journal, ++ new files under `docs/findings/`). + +**Read:** plan §0–§12 whole; `dn-autopilot-and-delegated-blessing` §1.2 non-goals 4/8 and the +`:102-104` out-of-scope clause, §2.4 whole, §2.8 whole, §2.9 invariants 5 and 7; +`finding-0263` whole; `.claude/hooks/_lib.py:126-233` (`_seg_match` / `glob_match` / +`matches_any` / `parse_front_matter` / `_scalar`), `:354-361` (`plan_write_scope`), `:455-475` +(`cmd_scope_check`'s widened allow-set); `scripts/board.py:25-70` (the `sys.path` idiom and +`_is_absent`); `scripts/capsule.py:1-90`; `tests/unit/test_capsule.py:405-431` (the AST +precedent); `docs/templates/build-plan.md:1-20,100-120`; `docs/templates/intent-capsule.md`; +`the-false-success-rule.md:15-70`; `pyproject.toml:103-135`. + +**Grounding confirmed before writing code:** + +- 137 plan directories; **118** carry a `**Touches stored data?**` line (the finding measured + 111 — this wave's own new plans account for the delta). `touches_stored_data: false` still + appears **zero** times. +- `## 7. Items` is the §7 heading form in **132** plans — one stable spelling, so a + `^##\s*7[.)]?\s` section locator is safe. +- `_lib.parse_front_matter` yields `[]` for `write_scope: []` (the flow-list branch) and `""` + for a key with no entries. Both must read as **absent** — UNDETERMINED, never PASS. + +**Design decisions taken before the first line of code (recorded so they are reviewable):** + +1. **The conjunction takes a keyed mapping, not a sequence.** `conjoin(dict[str, Result])` + returns PASS iff the key set is exactly `{P1..P5}` **and** all five are PASS. A sequence + would let `all(...)` be vacuously true over the empty list — Item 17's degenerate in its + purest form. Making the complete term set a *precondition* kills it structurally rather + than by a guard clause bolted on afterwards. +2. **P2 is checked in both glob directions**, both through `_lib.matches_any`: + (a) `matches_any(scope_entry, FORBIDDEN)` catches a concrete entry inside a forbidden tree + (`eval/foo.py`); (b) `matches_any(forbidden_witness, [scope_entry])` catches a scope glob + that *covers* a forbidden surface without naming it (`.claude/**`, `docs/**`, `**`). + Neither direction alone suffices — checked by hand against `glob_match`: the path + `.claude/**` is **not** matched by the pattern `.claude/hooks/**`, so only (b) sees it; + and the path `eval/foo.py` is matched by no witness, so only (a) sees it. +3. **The correction banner is sliced out of `__doc__`**, not duplicated into a constant, so + the docstring and the report output cannot drift (owner DRY rule). Extraction raises + loudly if the markers are missing rather than yielding an empty caveat — a silently + caveat-free report is exactly the false statement §2.4 warns about. +4. **P5 tolerates a trailing YAML `#` comment** on `session_budget` (`_lib._scalar` leaves it + glued on unquoted scalars, by deliberate design at `:218-233`). Rejecting legal YAML would + be enforcing a spelling rather than a property — Item 15's falsifier shape. The integer + itself is still exact. + +## 2026-07-27 — Items 14–17 complete; 109 tests green; mutation campaign 12/12 killed + +**Delivered** (both files new, both inside `write_scope`; nothing else in the tree was +touched — `git status` shows exactly `scripts/autopilot_eligibility.py`, +`tests/unit/test_autopilot_eligibility.py`, `docs/findings/finding-0271.md`, and this +journal): + +- `scripts/autopilot_eligibility.py` — `Result` (3-valued), `Term`, `Evaluation`, + `check_p1..check_p5`, `conjoin`, `report_line`, `diagnostics`, and the `check` / `report` + CLI. Stdlib + `_lib` only; imports `{__future__, argparse, enum, re, sys, dataclasses, + pathlib, _lib}` and nothing else. +- `tests/unit/test_autopilot_eligibility.py` — 109 tests. + +**Acceptance actually run, per item:** + +| item | acceptance | result | +|---|---|---| +| 14 | P1 FAIL on absolute / `../` / normalized-escape; PASS on ordinary + `**`; P2 FAIL on each of the five §2.4 members and on covering globs `.claude/**`, `docs/**`, `**`, `.claude/hooks/*.py`; PASS disjoint; matcher-identity + AST tests | green | +| 15 | PASS only for `no`/`No`/`No.`/`**No.**`/run-on; FAIL for `yes`, `Yes — …`, `Reads only.`, `No (reads the corpus)`, `No — reads the vault`, `Not directly.`; UNDETERMINED for absent / duplicated / zero-item; docstring banner asserted | green | +| 16 | P4 FAIL on all eight deny-list tokens incl. inside a fence; PASS clean; UNDETERMINED on absent/empty §7. P5 PASS `1`; FAIL `0`, `-1`, `1.5`, `one`, `1 2`; UNDETERMINED absent/`""`/`null`/`none`/`~` | green | +| 17 | all 3⁵ = **243** combinations enumerated — exactly **one** overall PASS, and **no** combination containing an `UNDETERMINED` passes; `check` exit 0 iff PASS; `report` one line, 145 chars for 20 globs (<200); AST invariant; read-only census | green | + +`uv run pytest tests/unit/test_autopilot_eligibility.py -q` → **109 passed in 0.33s**. + +**⚑ The predicate reddens on ABSENCE, not only on violation — the proof, by named test:** + +| absence | vacuously-true reading | named test | verdict | +|---|---|---|---| +| `write_scope` empty/absent | "every glob is inside the worktree" | `test_p1_degenerate_empty_scope_is_undetermined_not_pass` | UNDETERMINED | +| `write_scope` empty/absent | "scope ∩ forbidden = ∅" | `test_p2_degenerate_empty_scope_is_undetermined_not_pass` | UNDETERMINED | +| glued `#` entry (finding-0085) | intersection empty on a scope naming `eval/` | `test_p2_degenerate_glued_inline_comment_fails_and_names_finding_0085` | FAIL | +| zero `### Item ` headings | "every item carries the flag as no" | `test_p3_degenerate_zero_items_is_undetermined_not_pass` | UNDETERMINED | +| absent / duplicated flag line | — | `test_p3_undetermined_for_{an_item_with_no_flag_line,two_flag_lines_in_one_item}` | UNDETERMINED | +| §7 absent or item-less | "no acceptance step runs `deploy`" | `test_p4_degenerate_absent_or_empty_section_7_is_undetermined_not_pass` | UNDETERMINED | +| `session_budget` absent/`null` | — | `test_p5_degenerate_absent_empty_or_null_is_undetermined_not_pass` | UNDETERMINED | +| **the term set itself incomplete** | `all(...)` over an empty sequence | `test_conjunction_refuses_an_incomplete_or_wrong_term_set` | FAIL | +| all five UNDETERMINED | `all(t is not FAIL)` | `test_conjunction_over_all_243_term_combinations` | FAIL | + +The last two are the structural half: `conjoin` takes a **keyed mapping and requires the +complete `{P1..P5}` key set**, so a caller that computed nothing receives FAIL rather than a +vacuous PASS. `test_the_naive_conjunction_would_admit_thirty_two_combinations` pins the +arithmetic — the naive form admits 2⁵ = 32 combinations, `conjoin` admits exactly 1. + +**Mutation campaign — 12 mutants, 12 KILLED, 0 survivors** (harness: patch in place, run the +suite, restore; the restored file was `diff`-verified identical to the pre-campaign copy). + +| mutant | killed by | +|---|---| +| 14-m1 forbidden set omits `.claude/hooks/**` | `test_the_forbidden_set_is_the_verbatim_2_4_membership` | +| 14-m2 intersection compares literal strings, not globs | `…concrete_path_inside_a_forbidden_tree[eval/some/deep/new_file.py]` | +| 14-m2b covering direction dropped | `…covers_a_forbidden_surface_without_naming_it[**, .claude/**, docs/**]` (3) | +| 14-m3 / m3b empty scope → PASS (P1, then P2) | `…p1_degenerate_empty_scope…`, `test_diagnostics_name_every_non_pass_term_in_order`, `test_check_exits_zero_iff_overall_pass` | +| 15-m1 `startswith("no")` for exact equality | `…hedged_spelling_from_the_census[No (reads the corpus), No —…, No, but…, Not directly.]` (4) | +| 15-m2 zero-item case → PASS | `test_p3_degenerate_zero_items_is_undetermined_not_pass` | +| 15-m3 drop truncation at `**Parallelizable?**` | `…truncation_at_a_run_on_bolded_field_is_load_bearing` + 3 census fixtures | +| 16-m1 skip fenced blocks | `test_p4_degenerate_a_token_inside_a_fenced_block_still_fails` | +| 16-m2 empty §7 → PASS for P4 | `test_p4_degenerate_absent_or_empty_section_7…` | +| **17-m1 `all(t is not Result.FAIL)`** | `test_conjunction_over_all_243_term_combinations` + the 32-vs-1 arithmetic test | +| 17-m2 conjunction accepts an incomplete term set | `test_conjunction_refuses_an_incomplete_or_wrong_term_set` (all 5 params) | + +**Census (Item 17, read-only over all 137 `docs/build-plans/*/plan.md`):** + +``` +overall: {'pass': 13, 'fail': 124, 'undetermined': 0} + P1: {'pass': 137, 'fail': 0, 'undetermined': 0} + P2: {'pass': 100, 'fail': 37, 'undetermined': 0} + P3: {'pass': 32, 'fail': 80, 'undetermined': 25} + P4: {'pass': 104, 'fail': 27, 'undetermined': 6} + P5: {'pass': 137, 'fail': 0, 'undetermined': 0} +``` + +13 of 137 pass all five — **not** a constant function, so §8's *"fails its keep if the census +shows the predicate returns the same value for every real plan"* does not trip. No term is +constant-`UNDETERMINED`, so **Item 17's falsifier does not fire** and §10's first stop +condition is not met. `bp-120` — the real plan §2 item 4 named as the test target — passes all +five (`achievable: 3 globs; P1 pass P2 pass P3 pass P4 pass(lexical) P5 pass; …`). + +**Falsifiers, considered explicitly:** + +- **Item 14 — "the two matchers disagree."** *Did not fire.* Drilled as the plan specifies: + five real `write_scope` lists × an 11-path probe set, `_lib.matches_any` vs the predicate's + — identical verdicts on every pair (the predicate *is* `_lib.matches_any`, asserted by + identity and by an AST test that no local glob function exists). Plus the semantic bridge: + wherever the guard would let a real scope reach a forbidden witness, P2 must FAIL — it does. +- **Item 15 — "the regex rejects a correctly-written plan on a formatting detail."** + *Fires partially; does NOT disqualify.* Inspected every one of the 25 `UNDETERMINED`. + Nine plans fold the flag onto another field's bullet (`- **Invariant(s):** … **Touches + stored data?** no.`), which the bullet-anchored §6 regex misses; 5 of the 25 are + UNDETERMINED for that reason. §10's condition is *"dominated by"* — 5/25 = 20% is not + domination, and the folded form is not template-legal (`build-plans.md:111` gives the flag + its own bullet). The other 20: 5 plans predate §7 entirely, 1 uses bold-paragraph items and + carries the flag zero times, 14 genuinely omit it. ⚑ **The miss is safe by construction** — + a folded flag leaves its item with no bullet-anchored line, so the item is `UNDETERMINED` + and the plan FAILs; it can never produce a false PASS, not even on a folded `Yes`. I did + **not** widen the regex (§6 is the pinned authoritative form; §4 forbids silent + re-interpretation). **Filed `finding-0271`**, routed to the orchestrator with three options. +- **Item 16 — "P4 fires on a plan that merely mentions `deploy`."** *Fires, as predicted, and + is recorded rather than tuned away.* 27 of 137 plans FAIL P4. Most are true positives + (plans that genuinely shell out or deploy). The clearest false fire is **`bp-137` itself**: + its Item 16 enumerates the deny-list, so the plan that builds P4 fails P4 on all eight + tokens. Not disqualifying for the QoL plans §2.4 targets (spell check, a keybinding), so no + `spec-defect` is filed per Item 16's own condition; the limitation is stated in the tool's + docstring **and in its report output**, which is where §2.4 requires it. +- **Item 17 — "every real plan UNDETERMINED on the same term."** *Did not fire* (census above). + +**§3 Q3 obligation discharged:** the widened allow-set asymmetry (`cmd_scope_check` adds the +plan, its journal and `docs/findings/**`) is stated in the correction banner the `report` +subcommand emits, not only in a docstring a builder would read. + +**Green gate, every leg run separately:** + +| leg | result | +|---|---| +| `uv run ruff check .` | All checks passed! | +| `uv run mypy core agents eval ops scheduler scripts` | Success: no issues found in 263 source files | +| `uv run mypy` | **Found 69 errors** in 20 files (checked 564) — baseline **69**, asserted | +| `uv run python -m ops.type_gate` | exit 0; Tier-2 membership OK, bare-ignore scan OK | +| `uv run pytest -q` (CI form) | 4 failed, 2511 passed, 11 skipped, 21 deselected | + +⚑ The 4 failures are **pre-existing and environmental, proved twice.** (1) With this plan's +three new files moved out of the tree, the identical set fails. (2) Three of them +(`test_worktree_enforcement::test_{a,c,d}`) pass under `env -u OUROBOROS_HOOKS_OFF` — they are +the direct consequence of the owner's hook escape hatch armed tonight (`1c907f5`; +`scope-guard.sh:7` short-circuits on it), so `scope-guard` returns ALLOW where the test expects +DENY. CI does not export the variable and is unaffected. The fourth, +`test_handoff_availability::test_the_generator_reads_the_worktree_s_own_seat_not_the_main_checkout`, +fails under both and is a worktree-nesting artifact of running inside a worktree; it is outside +this plan's `write_scope` and untouched by it. `test_core_imports_nothing_outside_core` is the +standing finding-0105 deselect the CI form already carries. + +**Findings filed:** `finding-0271` (spec-defect → orchestrator, non-blocking) — P3's +bullet-anchored regex vs the folded-flag authoring form, with the 9-plan measurement, the +safety argument, and three options for the owner batched onto `finding-0263`. + +**Not done, and why:** nothing from §7 is outstanding. Deliberately not done, per §9: no +template edit, no normalization sweep of the historical plans, no `_lib` edit, no +prose-context heuristics in P4, no character cap on the capsule, no grant/secret/status flip. diff --git a/docs/findings/finding-0271.md b/docs/findings/finding-0271.md new file mode 100644 index 0000000..18524df --- /dev/null +++ b/docs/findings/finding-0271.md @@ -0,0 +1,114 @@ +--- +type: finding +id: finding-0271 +status: open +created: 2026-07-27 +updated: 2026-07-27 +links: + - docs/build-plans/bp-137/plan.md + - docs/findings/finding-0263.md + - docs/design-notes/dn-autopilot-and-delegated-blessing.md + - docs/templates/build-plan.md + - scripts/autopilot_eligibility.py +ftype: spec-defect +origin_plan: bp-137 +route: orchestrator +resolution: null +--- + +# P3's pinned regex is bullet-anchored, but 9 plans fold `**Touches stored data?**` onto another field's bullet — the flag is invisible to the check in 5 of the 25 plans it calls UNDETERMINED + +## What + +`bp-137` §6 pins P3's form (the `finding-0263` correction): + +> For each `### Item ` heading in the plan's §7, the item body must contain exactly one line +> matching `^\s*[-*]\s*\*\*Touches stored data\?\*\*\s*(?P.*)$` + +The regex is **anchored at the bullet**: the flag must open its own list item. That is exactly +`docs/templates/build-plan.md:111`'s form, and it is implemented verbatim in +`scripts/autopilot_eligibility.py`. But a second authoring form exists in the tree — the flag +appended to the end of *another* field's bullet: + +``` +- **Invariant(s):** none touched. **Touches stored data?** no. (bp-091:133) +- **Invariant(s):** read-only. **Touches stored data?** no. **Parallelizable?** no. (bp-072:182) + the env-only get_secret. **Touches stored data?** No (reads config…). (bp-067:223) +``` + +**Measured 2026-07-27** over all 137 `docs/build-plans/*/plan.md`, comparing the count of raw +`**Touches stored data?**` occurrences inside §7 against the count the pinned regex matches +inside `### Item ` bodies: + +| plan | raw in §7 | seen by the regex | P3 verdict | +|---|---|---|---| +| bp-035 | 3 | 2 | fail | +| bp-038 | 3 | 1 | undetermined | +| bp-067 | 3 | 0 | undetermined | +| bp-068 | 2 | 0 | undetermined | +| bp-072 | 5 | 0 | undetermined | +| bp-090 | 4 | 3 | fail | +| bp-091 | 3 | 1 | undetermined | +| bp-092 | 4 | 3 | fail | +| bp-137 | 5 | 4 | pass | + +Nine plans. `bp-137`'s own row is benign and worth naming so it is not read as a defect: its +fifth occurrence is a *quotation* of the flag inside Item 15's degenerate-input prose, not a +declaration, and the regex is right to ignore it. + +## Why it matters + +**It is safe by construction, which is why this is a `spec-defect` and not a `blocker`.** A +folded flag is never read as PASS-evidence. The item that carries it has no bullet-anchored +line, so that item is `UNDETERMINED`, and `UNDETERMINED` is absorbing under the conjunction — +the plan's overall verdict is `FAIL`. **The miss can only ever refuse; it can never produce a +false PASS**, not even on a folded `Yes`. That is the invariant-7 behaviour the whole plan is +built around, holding on a case its author did not anticipate. + +What it costs is *reach*, not safety: 5 of the 25 plans P3 currently calls `UNDETERMINED` are +undetermined because of the authoring form rather than because the author hedged. `bp-137` +Item 15's named falsifier is *"the pinned regex rejects a correctly-written new plan because +of a formatting detail the template permits — then the check is enforcing a spelling, not a +property, and it will train authors to game it."* + +**Verdict on the falsifier: it fires, but it does not disqualify.** The §10 stop-and-raise +condition is *"P3's `UNDETERMINED` set is **dominated** by template-legal shapes the regex +missed"*. It is not dominated — 5 of 25 (20%). The other 20 are: 5 plans with no §7 section at +all (`bp-000`–`bp-005`, which predate the template), 1 whose items are bold paragraphs rather +than `### Item ` headings and which carries the flag zero times (`bp-128`), and 14 whose items +genuinely carry no flag. And the folded form is **not** template-legal in the strict sense: +`docs/templates/build-plan.md:111` renders each field as its own bullet. So the build +continued rather than parking the criterion. + +The builder deliberately did **not** widen the regex. `bp-137` §6 is the authoritative pinned +form, and §4 states that any divergence from it is *"a `spec-defect`, never a silent +re-interpretation"*. Loosening a security-relevant predicate's parse on a builder's own +authority is exactly the move that rule forbids. + +## Re-entry condition + +**Not blocking. `bp-137` shipped against §6 as pinned.** The choice is the owner's, and it is +the *same* choice `finding-0263` already routes to him — batch it there: + +- **Option A — pin the authoring form.** Amend `docs/templates/build-plan.md:111` to state + that the flag must open its own bullet, and leave the regex as is. Cheapest; makes the + existing check correct-by-convention going forward; does not touch the 9 historical plans + (a normalization sweep remains `bp-137` §9 non-goal 1). +- **Option B — widen the regex** to `(?:^|\.\s|\s{2})\*\*Touches stored data\?\*\*\s*(...)$` + or similar, accepting a mid-line match. Reaches the 9 plans, but the "exactly one line per + item" rule that makes a duplicated flag `UNDETERMINED` becomes harder to state, and a + quotation of the flag in prose (as in `bp-137` itself) would start counting as a + declaration — trading a safe under-read for an unsafe over-read. +- **Option C — the front-matter key.** Subsumes this entirely, and is already + `finding-0263`'s parked design question (`touches_stored_data: false` as a real per-item + machine-read field). If the owner takes C, this finding closes with it. + +⚑ Whichever is chosen, the change lands in a **new plan**: `docs/templates/build-plan.md` is +outside `bp-137`'s `write_scope` and the design note is ratified and agent-immutable (A8). + +## Routing + +`spec-defect` → **orchestrator**. The implementation half was settled inside `bp-137` on the +builder's own authority (implement §6 verbatim, refuse rather than guess, record the census). +The template/regex half is a `design` question about an authoring convention and belongs in +the same owner batch as `finding-0263` and the `oq-0047` ftype ruling. From e8f4e58da89654d8d94a70c737c8c1651c1e8cee Mon Sep 17 00:00:00 2001 From: Alberto Serrano-Calva Date: Mon, 27 Jul 2026 20:13:54 -0400 Subject: [PATCH 3/3] fix(bp-137): renumber finding-0271 -> finding-0274, resolving a live id collision MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit bp-136 and bp-137 ran in parallel worktrees and BOTH independently minted a finding-0271 — two entirely different findings on the same number. Nothing in the tree allocates ids; each agent eyeballed the highest existing (finding-0270) and picked the next. bp-136 keeps 0271-0273 (it filed three, sequentially); this one takes 0274. Recorded because it is the exact defect bp-140 exists to remove, demonstrating itself on the night dn-typed-workflow-registry was ratified. We were lucky: the paths collided, so git would have conflicted loudly at merge. The silent case — two agents choosing different paths — is the one that corrupts quietly. --- docs/build-plans/bp-137/journal.md | 6 +++--- docs/findings/{finding-0271.md => finding-0274.md} | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) rename docs/findings/{finding-0271.md => finding-0274.md} (99%) diff --git a/docs/build-plans/bp-137/journal.md b/docs/build-plans/bp-137/journal.md index ab69d9a..6f6f5a9 100644 --- a/docs/build-plans/bp-137/journal.md +++ b/docs/build-plans/bp-137/journal.md @@ -84,7 +84,7 @@ precedent); `docs/templates/build-plan.md:1-20,100-120`; `docs/templates/intent- **Delivered** (both files new, both inside `write_scope`; nothing else in the tree was touched — `git status` shows exactly `scripts/autopilot_eligibility.py`, -`tests/unit/test_autopilot_eligibility.py`, `docs/findings/finding-0271.md`, and this +`tests/unit/test_autopilot_eligibility.py`, `docs/findings/finding-0274.md`, and this journal): - `scripts/autopilot_eligibility.py` — `Result` (3-valued), `Term`, `Evaluation`, @@ -175,7 +175,7 @@ five (`achievable: 3 globs; P1 pass P2 pass P3 pass P4 pass(lexical) P5 pass; a folded flag leaves its item with no bullet-anchored line, so the item is `UNDETERMINED` and the plan FAILs; it can never produce a false PASS, not even on a folded `Yes`. I did **not** widen the regex (§6 is the pinned authoritative form; §4 forbids silent - re-interpretation). **Filed `finding-0271`**, routed to the orchestrator with three options. + re-interpretation). **Filed `finding-0274`**, routed to the orchestrator with three options. - **Item 16 — "P4 fires on a plan that merely mentions `deploy`."** *Fires, as predicted, and is recorded rather than tuned away.* 27 of 137 plans FAIL P4. Most are true positives (plans that genuinely shell out or deploy). The clearest false fire is **`bp-137` itself**: @@ -210,7 +210,7 @@ fails under both and is a worktree-nesting artifact of running inside a worktree this plan's `write_scope` and untouched by it. `test_core_imports_nothing_outside_core` is the standing finding-0105 deselect the CI form already carries. -**Findings filed:** `finding-0271` (spec-defect → orchestrator, non-blocking) — P3's +**Findings filed:** `finding-0274` (spec-defect → orchestrator, non-blocking) — P3's bullet-anchored regex vs the folded-flag authoring form, with the 9-plan measurement, the safety argument, and three options for the owner batched onto `finding-0263`. diff --git a/docs/findings/finding-0271.md b/docs/findings/finding-0274.md similarity index 99% rename from docs/findings/finding-0271.md rename to docs/findings/finding-0274.md index 18524df..bcdab15 100644 --- a/docs/findings/finding-0271.md +++ b/docs/findings/finding-0274.md @@ -1,6 +1,6 @@ --- type: finding -id: finding-0271 +id: finding-0274 status: open created: 2026-07-27 updated: 2026-07-27