release: v0.17.1 — the evals run for the first time, and the tails go to zero

.claude-plugin/marketplace.json

@@ -1,4 +1,5 @@
 {
+  "$schema": "https://json.schemastore.org/claude-code-marketplace.json",
   "name": "agent-stack",
   "owner": {
     "name": "ssheleg",
@@ -11,7 +12,7 @@
       "displayName": "Agent Stack",
       "source": "./plugins/agent-stack",
       "description": "Four skills: agent-orchestrator — tool-calling loops, pipelines with checkpoints, provider routing with fallback, memory architecture, plus the wallet side of reselling LLM access; agent-evals — run/trace/thread evals, LLM judges, and fixtures grown from production; agent-interop — MCP servers and clients, A2A agent cards, the MCP Registry, and gateways; agent-harness — system prompts, tool shaping, workflow-vs-agent, and auditing an agent system.",
-      "version": "0.17.0",
+      "version": "0.17.1",
       "author": {
         "name": "ssheleg",
         "url": "https://x.com/sshlg93"

CHANGELOG.md

@@ -1,5 +1,46 @@
 # Changelog
 
+## v0.17.1 — the evals run for the first time, and the tails go to zero
+
+Wave-3 of the 2026-08-29 family audit (rows AST-05, AST-07, AST-08, AST-09, AST-10),
+plus the two board rows wave 2 filed (AST-A1, AST-B). All fixes and docs — no
+description or trigger changes, hence a patch.
+
+- **AST-05: the eval suite is executed for the first time.** `test/evals/RESULTS.md`
+  gains two dated rows — haiku and sonnet, 2026-08-31 — each a fresh blind subagent
+  per trigger query against the family's 28 skill descriptions, plus all three
+  scenarios scored line by line. Both models: 11/12 triggers. Scenario lines: 10/12
+  (haiku), 9/12 (sonnet). Each miss is named — haiku false-triggered `agent-harness`
+  on "rewrite this one system prompt so it sounds friendlier" (q09), sonnet answered
+  `none` on the Russian harness-audit query (q04) — and the Method section states the
+  protocol and its three limits rather than presenting the rates as clean-room.
+- **AST-B: the generator-evaluator citation lands where the doctrine lives.**
+  `agent-evals` §5a now cites Anthropic's *tuning a standalone evaluator to be
+  skeptical is more tractable than making a generator self-critical* (read
+  2026-08-30) beside the sentence making the same claim, marked as convergence
+  rather than invention — beside the first executed eval run, as the board row asked.
+- **AST-A1: the nine-subsystem coverage check is done and the row closes.** The ninth
+  subsystem the filing row could not name is **Automation**; all nine map to existing
+  doctrine with a `file:line` each, and the suspected identity/approval-policy gap
+  resolves as a split verdict — approval policy covered (`governance.md`),
+  authentication mechanics a named delegation (`layers.md:78-80`), not a hole. The
+  map lives in the board row (`docs/evidence/backlog.md`), not in a new reference.
+- **AST-07: the README stops counting its references.** "Twenty references" had
+  drifted to 24 actual files within a week of being written; the aggregate count is
+  dropped in favour of the per-skill counts that are still true, with the reason
+  stated in place.
+- **AST-10: the orchestrator's reference index lists all 11 references.** The three
+  memory rows — `memory-architecture.md`, `memory-lifecycle.md`,
+  `memory-landscape.md` — join the index table, each paraphrasing its own
+  "Load this when" line.
+- **AST-08: `license: MIT` in all four skill front matters** (was 1 of 4 —
+  `agent-evals` alone carried it). Every front matter re-checked with
+  `yaml.safe_load` after the edit.
+- **AST-09: `$schema` in both manifests** — `claude-code-plugin-manifest.json` for
+  `plugin.json`, `claude-code-marketplace.json` for `marketplace.json`, the two
+  schemastore addresses that resolve (the third candidate, `claude-code-plugin.json`,
+  is a 404 and two siblings currently point at it; referred, not copied).
+
 ## v0.17.0 — the descriptions stop colliding, and the claims get their dates
 
 Wave-2 of the 2026-08-29 family audit (rows AST-02, AST-03, AST-04, AST-06, AST-11), plus

README.md

@@ -33,8 +33,9 @@ Part of the [ssheleg skill family](https://github.com/ssheleg/sshlg-skills).
 
 Four skills — `agent-orchestrator` for wiring the loop, `agent-evals` for proving it
 behaves, `agent-interop` for everything it talks to outside its own process,
-`agent-harness` for what it is **told** — and twenty references they load on demand,
-plus one scanner.
+`agent-harness` for what it is **told** — and the references they load on demand,
+plus one scanner. The per-skill sections below carry their own counts; no total is
+stated here, because the last one drifted stale within a week of being written.
 
 **The orchestrator** (`SKILL.md`) — what the agent reads first:
 

docs/evidence/backlog.md

@@ -26,6 +26,8 @@ row per line, appended, so two runs closing two rows do not land on the same lin
 | AG-04 + AG-05 | **The audit document refused a score and computed one.** `agent-harness/references/audit.md:20-25` said *"A prioritized change plan … **Not a score**"* and *"pass/fail with a named failure condition beats a scalar that names no fix"*; `:114` then computed `P = blast × confidence / effort` and `:128` ordered the plan by it. The manifesto backs the refusal (`pod-manifesto/manifesto.md:424`, *"these axes are not a fake numerical score"*) and names FOUR axes at `:419-422` — Impact, **Irreversibility**, Uncertainty, **Coordination** — of which two appeared nowhere in the pack (`grep -ci irreversib` → **0**, `grep -ci coordinat` → **0**) while `effort`, a **cost**, had been substituted into their place. So the file held neither position. | 2026-08-18 manifesto-conformance audit, rows AG-04 and AG-05 | 3 | 2 | 2 | **3.0** | **closed 2026-08-20** — the position taken is the one the document already argued for: publish the axes, drop the arithmetic. §*Priority — four axes, and no scalar* declares `<!-- priority-axes: impact, irreversibility, uncertainty, coordination -->` and states each with its High·Medium·Low values, cited to `manifesto.md:419-422`. The ordering rule is **the first axis that separates two findings**, which is inspectable in the direction the formula was not: `3 × 1 / 3` and `1 × 1 / 1` both print **1**, so *harms a user, judgement-tier, its own project* and *annoys a maintainer, judgement-tier, an hour* arrived at one number and the inputs could not be recovered from it. `effort` keeps its job — sizing the plan's first three items — and loses its rank. `check_priority_axes_are_the_manifesto_s_and_carry_no_scalar` enforces the axis list as an **equality** (a fifth axis nobody sourced is as wrong as a missing one), each axis in the prose, the spelled count, both restored axes present somewhere in the pack, no prescribed scalar anywhere in it, and no plan ordered by one. Three negative self-tests in `validate.yml`, each watched at **exit 1**: `effort` substituted back for an axis, the formula reinstated as a standalone line, and `ordered by P`. A fourth plant run by hand — both restored axes renamed away across the pack — also refused. **One false positive was found by watching the check refuse a correct document**: the first draft matched the formula anywhere on a line, including the paragraph that *records its removal*. The guard now reads a whole-line prescription and not a citation, which is the umbrella's own rule — name a dead command, never claim it. |
 | AG-06 | **`test/plant_guard_test.py:34` leaked 8 nameless `tmpXXXXXXXX` trees per run and the gate said nothing.** Measured 2026-08-20: `find "$TMPDIR" -maxdepth 4 -type f -path '*/copy/sub/b.sh' \| wc -l` → **2568 before one run, 2576 after**. The fixture is byte-identical (md5 `623a086d10a04940573c31cbebb93e31`) in this repository and `seo-aeo-audit`. | AG-06, 2026-08-20 | 2 | 1 | 1 | **4.0** | **closed 2026-08-20 by PORTING, not reinventing** — `test/residue.py` comes from `make-skill`, which measured the same defect first and shipped the answer; a second implementation of one ledger is how two gates disagree about what a clean run leaves. Differences: the `agent-stack-test-` prefix, the suite it accounts for, and its opening paragraph. Every tree goes through `residue.workspace()`, a failing case KEEPS its tree (a plant is debugged by reading the copy it landed in), and both gate commands print a residue line — `npm test` now ends `residue: this run left nothing — 8 temp tree(s) created, 8 removed`, and `test/validate.py` carries the same tolerant import so the next thing that writes to `$TMPDIR` shows up in its own output. **The port alone was not enough, and watching it fail is what showed it:** the leak re-planted as a bare `tempfile.mkdtemp()` still PASSED and the ledger printed *"left nothing — 0 temp tree(s) created"* — a bypass reporting clean is indistinguishable from no leak. `check_temp_trees_go_through_the_residue_ledger` closes that: no `test/*.py` but `residue.py` may call `mkdtemp` or `TemporaryDirectory`, read from the source with comments and docstrings stripped so an explanation of the ban is not read as a violation. Both directions watched at **exit 1**, and one negative self-test each in `validate.yml`. |
 | AG-07 + AG-08 | **The ledger asserted a state that was false at HEAD.** Three sections of `docs/evidence/verification.md` were headed `(AG-0N, unreleased)` and carried *"No release: the version stays 0.11.1 and the CHANGELOG is untouched, so every row below is measured on the working tree rather than on a published artifact"* — while **v0.12.0 was tagged, on npm and in the CHANGELOG**. Under those paragraphs **35 rows** (15 + 11 + 9) all read `**verified**`, against the file's own opening at `:4-5`: *"A row sits at `never` until somebody has watched its check pass on the shipped artifact — not on a branch, not in a plan."* | 2026-08-18 manifesto-conformance audit, rows AG-07 and AG-08 | 3 | 1 | 1 | **6.0** | **closed 2026-08-20** — the rows are not re-graded on a promise: the gate was **re-run on the shipped artifact**. `git archive v0.12.0` (tag `7937c35`, released 2026-08-19T19:30:00Z, `npm view @ssheleg/agent-stack version` → `0.12.0`) into a clean tree, then `python3 test/validate.py` → `OK: agent-stack structurally valid (13 checks, 4 skill(s), v0.12.0)` exit **0**, and `python3 test/plant_guard_test.py` → `PASS: plant_guard — 9 cases` exit **0**. All three paragraphs now say what shipped and carry that receipt, and each records what it used to say. `check_the_ledger_matches_what_shipped` enforces both directions: a section declaring no release may carry **no** `verified` row, a section whose *"version stays X"* is behind the CHANGELOG's newest release is refused, and a section claiming `shipped in vX` needs a `## vX` in the CHANGELOG. Both plants watched at **exit 1**, the first naming the shipped defect exactly — *"declares no release and carries 15 row(s) marked **verified**"* — with a negative self-test each in `validate.yml`. |
+| AST-A1 | **Coverage check against OpenAI's nine-subsystem harness taxonomy.** Filed 2026-08-30 with the wave-2 description work; the suspected gap was identity/approval-policy. | 2026-08-30 harness-articles digest | 2 | 1 | 2 | **2.0** | **closed 2026-08-31** — the ninth subsystem the filing row could not name is **Automation**: the taxonomy as recovered via the `kenhuangus.substack.com` retelling (re-read 2026-08-31; `openai.com/index/harness-engineering` still 403) is Automation, Identity/Intent, Memory, Observability, Runtime Control, Evaluation, Scalability, Token Budget, Extensions. The map, one verdict per subsystem, each address checked at this commit: **Automation** (persistent shell, file-patch primitives) — covered as the worked kernel, `agent-harness/references/pi.md:33` (`read/write/edit/bash` and a loop) and `:191-198` (OpenShell, where execution actually runs), with the agent–computer interface in `agent-harness/references/tools.md`; **Identity/Intent** — split verdict, and it resolves the suspected gap: *approval policy* is covered — `agent-orchestrator/references/governance.md:52-56` (a deterministic limit or a human, never a classifier's confidence), `:24-37` (four boundaries with per-tool authorisation), `:75-91` (the audit row: who ran it, who changed the policy, which policy version) — while *authentication mechanics* (PKCE, workload federation) are a named delegation rather than a hole: `agent-harness/references/layers.md:78-80` (*identity and secrets — a credential store, never the loop*), with the MCP OAuth surface at `agent-interop/references/mcp.md:201-210`; **Memory** — `agent-orchestrator/references/memory-architecture.md`, `memory-lifecycle.md`, `memory-landscape.md`, plus `agent-orchestrator/SKILL.md:225` (§7); **Observability** — `agent-evals/SKILL.md:48-50` (traces durable and queryable as the precondition), `:273` (§7 what to instrument first), `agent-orchestrator/SKILL.md:264` (§9); **Runtime Control** — `agent-orchestrator/references/governance.md:127-138` (a sandbox protects the host; egress allowlists; credentials never enter) and `agent-harness/references/layers.md:59-77` (why the boundary is the environment's, and how deciding the layer decides the audit); **Evaluation** — `agent-evals/SKILL.md` end to end, the in-loop half at §5a (`:164`), the audit half `agent-harness/references/audit.md`; **Scalability** — `agent-orchestrator/references/runtime.md` (thread lifecycles: checkpoint every iteration, interrupt/resume, double-texting, schedules), `references/graph-engineering.md` (parallel layers and what a host executes on fan-out), `references/provider-lifecycle.md` (workspace lifecycle, fleet budgets); **Token Budget** — `agent-orchestrator/references/context-engineering.md` (compaction ladder, token estimation, circuit breaker), the money meter in `references/llm-proxy-billing.md`; **Extensions** — `agent-harness/references/pi-sdk.md:247` (the extension seams), the outward protocol in `agent-interop/references/mcp.md`, and skill construction deliberately NOT covered — that is `make-skill`, the boundary both descriptions already draw. Verdict: **nine of nine either covered with an address or delegated with the delegation named; no reference gains a rewrite from this row.** The map lives here per this board's convention — the row is the record, and a doctrine home for it would be a tenth document pointing at nine existing ones. |
+| AST-B | **The generator-evaluator claim in `agent-evals` matched Anthropic's harness-design article and did not cite it.** | 2026-08-30 harness-articles digest | 3 | 1 | 1 | **6.0** | **closed 2026-08-31** — the citation lands where the doctrine lives, `agent-evals/SKILL.md` §5a: Anthropic's *tuning a standalone evaluator to be skeptical is more tractable than making a generator self-critical* (`anthropic.com/engineering/harness-design-long-running-apps`, read 2026-08-30) now sits beside the sentence that makes the same claim — the convergence depends on the checker, not on the branches grading themselves — dated, and marked as convergence rather than invention. Landed beside the first executed eval run (AST-05, `test/evals/RESULTS.md`), as the filing row asked. `check_one_home_per_fact` stays green on the quoted sentence: a citation, not a restatement. |
 | AG-09 | **Three of four skill descriptions were past the house working limit**, one with **five** characters of headroom before the hard 1024 cap: `agent-orchestrator/SKILL.md:3` at **1019**, `agent-evals` **986**, `agent-interop` **983** (measured by `make-skill/scripts/audit_skill.py --house`, whose `DESC_TARGET` is 970). A description at 99% of cap cannot absorb the *and not for X* clause the next near-miss neighbour requires — which is the clause that stops a sibling stealing its triggers. | AG-09, 2026-08-20 | 2 | 1 | 1 | **4.0** | **closed 2026-08-20** — prose compressed, **no trigger phrase touched**: quoted-trigger counts before and after are 19→19, 12→12 and 14→14, `lost: []` in each case. Now 964 / 963 / 970 against the 970 limit, and `audit_skill.py --house` reports `0 GAP` for all four skills (`agent-harness` was already at 916). The limit is also a **gate** now, not only an external auditor's opinion: `test/validate.py` refuses anything past `DESC_WORKING_LIMIT = 970`, sourced by name to make-skill's `DESC_TARGET` rather than re-decided here. Watched at **exit 1** on a description padded to 1014 chars — under the hard cap, over the working limit, which is the band the old check could not see — plus a negative self-test in `validate.yml`. |
 
 ## Open, and why
@@ -34,6 +36,4 @@ row per line, appended, so two runs closing two rows do not land on the same lin
 |---|---|---|
 | AG-06a | **A temp tree created outside `test/` is still invisible.** `check_temp_trees_go_through_the_residue_ledger` reads this repository's own suite. A tree made by anything else — a script under `plugins/`, a future helper outside `test/` — is not accounted for, and `residue.report()` counts only what `workspace()` handed out. The mechanism catches a *reviewed* bypass; it does not observe `$TMPDIR`. Observing it would mean a run claiming ownership of directories it cannot prove it made, which is the reaping mistake `agent-sync` refuses by design. | Open: the ledger reads this repository's own suite, so a tree made anywhere else is still unattributable. |
 | AG-06b | **The 2592 existing trees are reported and left alone.** `find "$TMPDIR" -maxdepth 4 -type f -path '*/copy/sub/b.sh' \\| wc -l` → **2592** on 2026-08-20, of which **16 were added by this run's own verification** — one run of the leaking fixture before the port, and one of the v0.12.0 archive's copy of it. They are plain `tmpXXXXXXXX` and indistinguishable from every other program's, so they are named rather than swept. The identical fixture still leaks in **`seo-aeo-audit`**, which another agent holds; referred, not edited. | Open by decision: state this run cannot prove is its own is reported, never deleted. |
-| AST-A1 | **Coverage check against OpenAI's nine-subsystem harness taxonomy.** OpenAI's harness-engineering article (`openai.com/index/harness-engineering`, read 2026-08-30) enumerates nine subsystems a harness owns — identity/intent, memory, observability, runtime control, evaluation, scalability, token budget, extensions among them. `agent-harness` now cites the term (v0.17.0) but nobody has walked our references against that list; the suspected gap is **identity / approval policy**, which `references/layers.md` currently waves at as "usually somebody else's job". The check is a mapping table with a `file:line` per subsystem, or a named refusal per subsystem we deliberately do not cover. | Open: filed 2026-08-30 with the wave-2 description work; the mapping is a session of reading, not a line edit. |
-| AST-B | **The generator-evaluator claim in `agent-evals` matches Anthropic's harness-design article and does not cite it.** Anthropic (read 2026-08-30): "tuning a standalone evaluator to be skeptical is more tractable than making a generator self-critical" — the doctrine `agent-evals` already carries. Add the dated citation where the claim lives, so the next reader can tell doctrine that converged with the field from doctrine invented here. | Open: filed 2026-08-30; deferred to the evals-day wave (AST-05) so the citation lands beside the first executed eval run rather than alone. |
 | AG-04a | **This board's own priority is a scalar.** The header computes `P = blast × (1 + age_runs) / effort` while `audit.md` now argues that publishing axes beats multiplying them. The two are not the same claim — the audit is doctrine this pack ships to users about somebody else's system, the board is one repository's own triage — but the tension is real and it is not resolved by saying so. The method's home is the umbrella board (`sshlg-skills/docs/evidence/backlog.md`), which another agent holds; the check added here deliberately scopes to documents under `plugins/`, so it neither polices nor blesses this file. Referred to the umbrella. | Open: the board's formula and `audit.md` disagree, and neither has been chosen. |