diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 25700db..db1131f 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -11,7 +11,7 @@ "displayName": "Agent Stack", "source": "./plugins/agent-stack", "description": "Four skills: agent-orchestrator — tool-calling loops, pipelines with checkpoints, provider routing with fallback, memory architecture, plus the wallet side of reselling LLM access; agent-evals — run/trace/thread evals, LLM judges, and fixtures grown from production; agent-interop — MCP servers and clients, A2A agent cards, the MCP Registry, and gateways; agent-harness — system prompts, tool shaping, workflow-vs-agent, and auditing an agent system.", - "version": "0.16.1", + "version": "0.17.0", "author": { "name": "ssheleg", "url": "https://x.com/sshlg93" diff --git a/CHANGELOG.md b/CHANGELOG.md index e37f595..f143aac 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,48 @@ # Changelog +## v0.17.0 — the descriptions stop colliding, and the claims get their dates + +Wave-2 of the 2026-08-29 family audit (rows AST-02, AST-03, AST-04, AST-06, AST-11), plus +the harness-engineering externals read 2026-08-30. + +- **AST-02: `agent-orchestrator` gives "checker node" back to `agent-evals`.** The quoted + trigger appeared verbatim in both descriptions, so the router had two skills advertising + one phrase. `agent-evals` owns it; the orchestrator's description already carries "a + checker before convergence" in prose, which is the claim it actually makes. +- **AST-11: the single-word triggers "agent" / "агент" narrow to "agent system" / + "агентная система".** A one-word trigger matching every sentence with the word *agent* + in it is how a skill teaches the router to route around the boundary its own description + draws ("Not for a single LLM call in a script"). The multiword triggers carry the load. +- **`agent-harness` advertises "ReAct loop" and "react pattern" instead of bare "ReAct"** — + a phrase the umbrella's deferred hook trigger can carry without firing on every mention + of the React framework. The body prose still explains ReAct by its own name. +- **AST-06: the harness↔orchestrator boundary names its four seams instead of claiming + one.** `agent-harness` §Boundaries said the two skills "meet in one place" while the + pack actually crosses at four: orchestrator §3 → harness `tools.md` (describing a tool + vs assembling the list), orchestrator §10 → harness `system-prompt.md` (what the prompt + says vs rebuilding it per request), harness static-or-dynamic → orchestrator + `graph-engineering.md`, and `context-engineering.md` (compaction) vs `system-prompt.md` + (filling). The rewrite states explicitly that shape-of-the-work doctrine has ONE home — + `agent-orchestrator/references/graph-engineering.md` — and the harness intro stops + claiming "the shape of the work" as its own ground, which was the sentence that made + both skills owners of one table. +- **AST-03/AST-04: `agent-interop`'s undated external claims get their dates.** "Moved in + the last twelve months", "SDKs and blog posts still document v0.3" and "its own overview + page still introduces `binds`" sat undated inside the very section whose rule is *a + protocol claim without a date is a guess* — all three now anchor to **2026-08-13**, the + stamp every reference in the skill already carries. The neighbourhood table gains a + "Verdicts as of 2026-08-13" line, since `test/validate.py` gates stamps only under + `references/` and the table lives in SKILL.md. +- **`agent-harness` cites the outside term for its ground: harness engineering.** OpenAI's + harness-engineering article and Anthropic's harness-design article (both read + 2026-08-30) name the layer this skill covers, with the measured leverage — ARC-AGI-3 + harness-level changes moved a fixed model 13.3%→38.3% at a sixth of the tokens (as + reported 2026-08-30). "harness engineering" joins the description triggers: users will + say it. The deeper work is filed, not done: board rows **AST-A1** (coverage check + against OpenAI's nine-subsystem taxonomy, identity/approval policy the suspected gap) + and **AST-B** (the dated generator-evaluator citation in `agent-evals`, deferred to + evals day). + ## v0.16.1 — the installers refuse the shadow, and the pack stops mis-selling itself - **Both installers refuse to write plain copies over an installed plugin.** The family diff --git a/docs/evidence/backlog.md b/docs/evidence/backlog.md index fa82aa7..3050258 100644 --- a/docs/evidence/backlog.md +++ b/docs/evidence/backlog.md @@ -34,4 +34,6 @@ row per line, appended, so two runs closing two rows do not land on the same lin |---|---|---| | AG-06a | **A temp tree created outside `test/` is still invisible.** `check_temp_trees_go_through_the_residue_ledger` reads this repository's own suite. A tree made by anything else — a script under `plugins/`, a future helper outside `test/` — is not accounted for, and `residue.report()` counts only what `workspace()` handed out. The mechanism catches a *reviewed* bypass; it does not observe `$TMPDIR`. Observing it would mean a run claiming ownership of directories it cannot prove it made, which is the reaping mistake `agent-sync` refuses by design. | Open: the ledger reads this repository's own suite, so a tree made anywhere else is still unattributable. | | AG-06b | **The 2592 existing trees are reported and left alone.** `find "$TMPDIR" -maxdepth 4 -type f -path '*/copy/sub/b.sh' \\| wc -l` → **2592** on 2026-08-20, of which **16 were added by this run's own verification** — one run of the leaking fixture before the port, and one of the v0.12.0 archive's copy of it. They are plain `tmpXXXXXXXX` and indistinguishable from every other program's, so they are named rather than swept. The identical fixture still leaks in **`seo-aeo-audit`**, which another agent holds; referred, not edited. | Open by decision: state this run cannot prove is its own is reported, never deleted. | +| AST-A1 | **Coverage check against OpenAI's nine-subsystem harness taxonomy.** OpenAI's harness-engineering article (`openai.com/index/harness-engineering`, read 2026-08-30) enumerates nine subsystems a harness owns — identity/intent, memory, observability, runtime control, evaluation, scalability, token budget, extensions among them. `agent-harness` now cites the term (v0.17.0) but nobody has walked our references against that list; the suspected gap is **identity / approval policy**, which `references/layers.md` currently waves at as "usually somebody else's job". The check is a mapping table with a `file:line` per subsystem, or a named refusal per subsystem we deliberately do not cover. | Open: filed 2026-08-30 with the wave-2 description work; the mapping is a session of reading, not a line edit. | +| AST-B | **The generator-evaluator claim in `agent-evals` matches Anthropic's harness-design article and does not cite it.** Anthropic (read 2026-08-30): "tuning a standalone evaluator to be skeptical is more tractable than making a generator self-critical" — the doctrine `agent-evals` already carries. Add the dated citation where the claim lives, so the next reader can tell doctrine that converged with the field from doctrine invented here. | Open: filed 2026-08-30; deferred to the evals-day wave (AST-05) so the citation lands beside the first executed eval run rather than alone. | | AG-04a | **This board's own priority is a scalar.** The header computes `P = blast × (1 + age_runs) / effort` while `audit.md` now argues that publishing axes beats multiplying them. The two are not the same claim — the audit is doctrine this pack ships to users about somebody else's system, the board is one repository's own triage — but the tension is real and it is not resolved by saying so. The method's home is the umbrella board (`sshlg-skills/docs/evidence/backlog.md`), which another agent holds; the check added here deliberately scopes to documents under `plugins/`, so it neither polices nor blesses this file. Referred to the umbrella. | Open: the board's formula and `audit.md` disagree, and neither has been chosen. | diff --git a/docs/evidence/verification.md b/docs/evidence/verification.md index 1219ad4..67a9c14 100644 --- a/docs/evidence/verification.md +++ b/docs/evidence/verification.md @@ -10,6 +10,22 @@ This file exists because its absence read as zero exposure. `sshlg-skills` board --- +## Shipped state — v0.17.0 (2026-08-30) + +Measured on the release-candidate tree before the tag exists. Wave-2 of the 2026-08-29 +family audit: AST-02/03/04/06/11 plus the harness-engineering externals read 2026-08-30. + +| REQ | What ships | How it was confirmed | Confirmed | +|---|---|---|---| +| R-01 | AST-02: the quoted trigger "checker node" lives in exactly one description | `grep -rn '"checker node"' plugins/agent-stack/skills/*/SKILL.md` → one hit, `agent-evals/SKILL.md:12`; `grep -c 'checker before' agent-orchestrator/SKILL.md` → 4, so the orchestrator still states its own claim in prose | **observed** | +| R-02 | AST-11: orchestrator's single-word triggers narrowed | description now carries `"agent system"` and `"агентная система"` (SKILL.md:10,12); neither bare `"agent"` nor bare `"агент"` remains as a quoted trigger there | **observed** | +| R-03 | ReAct advertised as a phrase, not a bare word | `agent-harness/SKILL.md:11` carries `"ReAct loop", "react pattern"`; the only quoted `"ReAct"` trigger is gone, so the umbrella hook can mirror the phrase without firing on the React framework | **observed** | +| R-04 | AST-06: the harness↔orchestrator boundary enumerates its four seams and names the one home | §Boundaries lists tools.md↔§3, system-prompt.md↔§10, static-or-dynamic→graph-engineering.md, context-engineering.md↔system-prompt.md, and states shape-of-the-work doctrine has ONE home (`agent-orchestrator/references/graph-engineering.md`); the intro no longer claims "the shape of the work"; `check_one_home_per_fact` (shingle floor 20) passes on the rewrite | **observed** | +| R-05 | AST-03/04: interop's external claims carry dates | "twelve months before **2026-08-13**" anchors the moved-spec list; the v0.3 and `binds` "still" claims each end "as of 2026-08-13"; the neighbourhood table opens "**Verdicts as of 2026-08-13**" | **observed** | +| R-06 | Harness engineering cited as the outside term, dated, with the deferred work filed | intro cites OpenAI's and Anthropic's articles (both read 2026-08-30) and the ARC-AGI-3 result 13.3%→38.3% at a sixth of the tokens (as reported 2026-08-30); `"harness engineering"` joins the triggers; board rows AST-A1 and AST-B filed in `docs/evidence/backlog.md` | **observed** | +| R-07 | Every description stays inside the working limit after the edits | `python3 test/validate.py` → `OK: agent-stack structurally valid (13 checks = 9 named + 4 per-skill, 4 skill(s), v0.17.0)`; measured 967 / 963 / 970 / 961 chars against the 970 limit; `audit_skill.py --house` → `0 GAP, 14 PASS` for each of the three edited skills | **observed** | +| R-08 | The whole gate is green on the bumped tree | `npm test` → validator OK line above, `PASS: plant_guard — 9 cases`, `PASS: installer — 11 case(s)`, both residue lines "left nothing" | **observed** | + ## Shipped state — v0.16.1 (2026-08-29) Measured on the release-candidate tree before the tag exists. The wave-1.5 rollout: diff --git a/package.json b/package.json index 3175a03..b111cab 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@ssheleg/agent-stack", - "version": "0.16.1", + "version": "0.17.0", "scripts": { "test": "python3 test/validate.py && python3 test/plant_guard_test.py && node test/installer_test.js" }, diff --git a/plugins/agent-stack/.claude-plugin/plugin.json b/plugins/agent-stack/.claude-plugin/plugin.json index eed29ca..115336f 100644 --- a/plugins/agent-stack/.claude-plugin/plugin.json +++ b/plugins/agent-stack/.claude-plugin/plugin.json @@ -2,7 +2,7 @@ "name": "agent-stack", "displayName": "Agent Stack", "description": "Four skills: agent-orchestrator — tool-calling loops, pipelines with checkpoints, provider routing with fallback, memory architecture, plus the wallet side of reselling LLM access; agent-evals — run/trace/thread evals, LLM judges, and fixtures grown from production; agent-interop — MCP servers and clients, A2A agent cards, the MCP Registry, and gateways; agent-harness — system prompts, tool shaping, workflow-vs-agent, and auditing an agent system.", - "version": "0.16.1", + "version": "0.17.0", "author": { "name": "ssheleg", "url": "https://x.com/sshlg93" diff --git a/plugins/agent-stack/skills/agent-harness/SKILL.md b/plugins/agent-stack/skills/agent-harness/SKILL.md index 391ceac..020ac66 100644 --- a/plugins/agent-stack/skills/agent-harness/SKILL.md +++ b/plugins/agent-stack/skills/agent-harness/SKILL.md @@ -8,7 +8,8 @@ description: >- instead of a score, plus a scanner. Carries Pi as a worked kernel implementation — SDK, RPC and extension seams — for embedding or extending a harness. Triggers - "system prompt", "tool description", "agent picks the wrong tool", "agent loops forever", "prompt - engineering", "ReAct", "workflow or agent", "static or dynamic", "audit this agent", + engineering", "harness engineering", "ReAct loop", "react pattern", "workflow or agent", + "static or dynamic", "audit this agent", "embed an agent", "agent SDK", "Pi harness", "системный промпт", "агент не вызывает тул", "аудит агента", "встроить агента". Not for the loop's plumbing, its evals, or its protocols — those are siblings. @@ -18,7 +19,13 @@ description: >- `agent-orchestrator` wires the loop. `agent-evals` proves it behaves. `agent-interop` gets it talking to other processes. **This skill is the layer between them and the model: the -prompt, the tools, and the shape of the work.** +prompt, the tools, and the choice of technique.** The outside term for this ground is +**harness engineering** — OpenAI's article of that name (`openai.com/index/harness-engineering`, +read 2026-08-30) and Anthropic's harness-design guidance +(`anthropic.com/engineering/harness-design-long-running-apps`, read 2026-08-30) both name +this same layer, and its leverage is measured: on ARC-AGI-3, harness-level changes alone +moved a fixed model from 13.3% to 38.3% while spending a sixth of the tokens (as reported +2026-08-30). It runs in both directions. Building one and auditing one are the same checklist read forwards and backwards, which is why they live together here. @@ -146,10 +153,22 @@ is then unfalsifiable — including this audit. Say so first, and make it the fi **Against `agent-orchestrator`.** That skill owns the loop's *plumbing*: iteration guards, trimming, sub-agent dispatch, provider routing, memory layers, checkpoints. This one owns -what the model is *told*. They meet in one place: -`agent-orchestrator/references/context-engineering.md` covers **compaction** — what to drop -when the window fills — while this skill's `system-prompt.md` covers what to put there in -the first place. Filling and emptying, two files. +what the model is *told*. They meet at four seams, each crossing in exactly one place: + +1. **Describing a tool** so the model picks the right one is this skill's + `references/tools.md`; assembling the tool *list* per request from capability flags is + the orchestrator's §3, which points here for the wording. +2. **What the prompt says** — altitude, vocabulary, enumerated statuses — is this skill's + `references/system-prompt.md`; *rebuilding* that prompt per request, in the same pass + as the tools, is the orchestrator's §10. +3. **The shape of the work** — the static/dynamic table, the fake-edge test, the checker + before a convergence — has ONE home, and it is not here: + `agent-orchestrator/references/graph-engineering.md`. This skill's static-or-dynamic + section stops at the decision and links there for the model. +4. **The context window:** `agent-orchestrator/references/context-engineering.md` covers + **compaction** — what to drop when the window fills — while this skill's + `system-prompt.md` covers what to put there in the first place. Filling and emptying, + two files. **Against `agent-evals`.** That skill measures whether an agent behaves, from execution records. This one reviews how it was *built*, from its source and prompts. An audit that diff --git a/plugins/agent-stack/skills/agent-interop/SKILL.md b/plugins/agent-stack/skills/agent-interop/SKILL.md index 5da75ff..6ce0cc0 100644 --- a/plugins/agent-stack/skills/agent-interop/SKILL.md +++ b/plugins/agent-stack/skills/agent-interop/SKILL.md @@ -24,19 +24,20 @@ registry, and the gateway between them. ## Rule zero — a protocol claim without a date is a guess -Every one of these specifications moved in the last twelve months, and each moved in a way -that silently breaks code written against the previous revision. Concretely, and each one -measured against the live specification rather than recalled: +Every one of these specifications moved in the twelve months before **2026-08-13** — the +date every reference below was read — and each moved in a way that silently breaks code +written against the previous revision. Concretely, and each one measured on that date +against the live specification rather than recalled: - MCP replaced the `initialize` handshake with `server/discover` and made the protocol **stateless**; a client that opens with `initialize` is speaking a revision that is on its way out. - MCP **deprecated `sampling`, `roots` and `logging`** — three things an older model will reach for first, because for a year they were the interesting part of the client side. -- A2A renamed its wire surface between v0.x and v1.0, and SDKs and blog posts still document - v0.3. +- A2A renamed its wire surface between v0.x and v1.0, and SDKs and blog posts still + documented v0.3 as of 2026-08-13. - agentgateway **deprecated `binds`** in favour of `gateways` — while its own overview page - still introduces `binds` as a core concept. + still introduced `binds` as a core concept as of 2026-08-13. So: **every reference in this skill opens with a `**Spec pinned:**` line**, and `test/validate.py` fails the build without one. That is a mechanical check, not an @@ -119,6 +120,8 @@ it); prompt wording; a single LLM call in a script. ## The neighbourhood — one verdict each Named so an agent stops guessing, with the verdict stated rather than implied. +**Verdicts as of 2026-08-13**, the day this skill's references were pinned — this ground +moves, so re-verify anything marked *watch* before building on it. | Thing | What it actually is | Verdict | |---|---|---| diff --git a/plugins/agent-stack/skills/agent-orchestrator/SKILL.md b/plugins/agent-stack/skills/agent-orchestrator/SKILL.md index 400f901..b4c6db1 100644 --- a/plugins/agent-stack/skills/agent-orchestrator/SKILL.md +++ b/plugins/agent-stack/skills/agent-orchestrator/SKILL.md @@ -7,9 +7,9 @@ description: >- architecture, retrieval and decay, context budgets, sub-agent coordination, error hierarchies; the work as a graph — parallel layers, fake edges, a checker before convergence; for resale: tiered wallets, one markup boundary, two-phase commit across database and provider API, - spend-delta polling, budget and loop guards, per-tenant keys. Triggers - "agent", + spend-delta polling, budget and loop guards, per-tenant keys. Triggers - "agent system", "orchestrator", "tool calling", "sub-agent", "LLM router", "fallback chain", "human in the - loop", "memory layer", "LLM billing", "token wallet", "checker node", "агент", "оркестратор", + loop", "memory layer", "LLM billing", "token wallet", "агентная система", "оркестратор", "суб-агент", "роутер моделей", "человек в цикле", "слой памяти", "биллинг LLM", "граф задач". Not for a single LLM call in a script, or prompt wording. ---