Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .claude-plugin/marketplace.json
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
"displayName": "Agent Stack",
"source": "./plugins/agent-stack",
"description": "Four skills: agent-orchestrator — tool-calling loops, pipelines with checkpoints, provider routing with fallback, memory architecture, plus the wallet side of reselling LLM access; agent-evals — run/trace/thread evals, LLM judges, and fixtures grown from production; agent-interop — MCP servers and clients, A2A agent cards, the MCP Registry, and gateways; agent-harness — system prompts, tool shaping, workflow-vs-agent, and auditing an agent system.",
"version": "0.16.1",
"version": "0.17.0",
"author": {
"name": "ssheleg",
"url": "https://x.com/sshlg93"
Expand Down
43 changes: 43 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,48 @@
# Changelog

## v0.17.0 — the descriptions stop colliding, and the claims get their dates

Wave-2 of the 2026-08-29 family audit (rows AST-02, AST-03, AST-04, AST-06, AST-11), plus
the harness-engineering externals read 2026-08-30.

- **AST-02: `agent-orchestrator` gives "checker node" back to `agent-evals`.** The quoted
trigger appeared verbatim in both descriptions, so the router had two skills advertising
one phrase. `agent-evals` owns it; the orchestrator's description already carries "a
checker before convergence" in prose, which is the claim it actually makes.
- **AST-11: the single-word triggers "agent" / "агент" narrow to "agent system" /
"агентная система".** A one-word trigger matching every sentence with the word *agent*
in it is how a skill teaches the router to route around the boundary its own description
draws ("Not for a single LLM call in a script"). The multiword triggers carry the load.
- **`agent-harness` advertises "ReAct loop" and "react pattern" instead of bare "ReAct"** —
a phrase the umbrella's deferred hook trigger can carry without firing on every mention
of the React framework. The body prose still explains ReAct by its own name.
- **AST-06: the harness↔orchestrator boundary names its four seams instead of claiming
one.** `agent-harness` §Boundaries said the two skills "meet in one place" while the
pack actually crosses at four: orchestrator §3 → harness `tools.md` (describing a tool
vs assembling the list), orchestrator §10 → harness `system-prompt.md` (what the prompt
says vs rebuilding it per request), harness static-or-dynamic → orchestrator
`graph-engineering.md`, and `context-engineering.md` (compaction) vs `system-prompt.md`
(filling). The rewrite states explicitly that shape-of-the-work doctrine has ONE home —
`agent-orchestrator/references/graph-engineering.md` — and the harness intro stops
claiming "the shape of the work" as its own ground, which was the sentence that made
both skills owners of one table.
- **AST-03/AST-04: `agent-interop`'s undated external claims get their dates.** "Moved in
the last twelve months", "SDKs and blog posts still document v0.3" and "its own overview
page still introduces `binds`" sat undated inside the very section whose rule is *a
protocol claim without a date is a guess* — all three now anchor to **2026-08-13**, the
stamp every reference in the skill already carries. The neighbourhood table gains a
"Verdicts as of 2026-08-13" line, since `test/validate.py` gates stamps only under
`references/` and the table lives in SKILL.md.
- **`agent-harness` cites the outside term for its ground: harness engineering.** OpenAI's
harness-engineering article and Anthropic's harness-design article (both read
2026-08-30) name the layer this skill covers, with the measured leverage — ARC-AGI-3
harness-level changes moved a fixed model 13.3%→38.3% at a sixth of the tokens (as
reported 2026-08-30). "harness engineering" joins the description triggers: users will
say it. The deeper work is filed, not done: board rows **AST-A1** (coverage check
against OpenAI's nine-subsystem taxonomy, identity/approval policy the suspected gap)
and **AST-B** (the dated generator-evaluator citation in `agent-evals`, deferred to
evals day).

## v0.16.1 — the installers refuse the shadow, and the pack stops mis-selling itself

- **Both installers refuse to write plain copies over an installed plugin.** The family
Expand Down
2 changes: 2 additions & 0 deletions docs/evidence/backlog.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,4 +34,6 @@ row per line, appended, so two runs closing two rows do not land on the same lin
|---|---|---|
| AG-06a | **A temp tree created outside `test/` is still invisible.** `check_temp_trees_go_through_the_residue_ledger` reads this repository's own suite. A tree made by anything else — a script under `plugins/`, a future helper outside `test/` — is not accounted for, and `residue.report()` counts only what `workspace()` handed out. The mechanism catches a *reviewed* bypass; it does not observe `$TMPDIR`. Observing it would mean a run claiming ownership of directories it cannot prove it made, which is the reaping mistake `agent-sync` refuses by design. | Open: the ledger reads this repository's own suite, so a tree made anywhere else is still unattributable. |
| AG-06b | **The 2592 existing trees are reported and left alone.** `find "$TMPDIR" -maxdepth 4 -type f -path '*/copy/sub/b.sh' \\| wc -l` → **2592** on 2026-08-20, of which **16 were added by this run's own verification** — one run of the leaking fixture before the port, and one of the v0.12.0 archive's copy of it. They are plain `tmpXXXXXXXX` and indistinguishable from every other program's, so they are named rather than swept. The identical fixture still leaks in **`seo-aeo-audit`**, which another agent holds; referred, not edited. | Open by decision: state this run cannot prove is its own is reported, never deleted. |
| AST-A1 | **Coverage check against OpenAI's nine-subsystem harness taxonomy.** OpenAI's harness-engineering article (`openai.com/index/harness-engineering`, read 2026-08-30) enumerates nine subsystems a harness owns — identity/intent, memory, observability, runtime control, evaluation, scalability, token budget, extensions among them. `agent-harness` now cites the term (v0.17.0) but nobody has walked our references against that list; the suspected gap is **identity / approval policy**, which `references/layers.md` currently waves at as "usually somebody else's job". The check is a mapping table with a `file:line` per subsystem, or a named refusal per subsystem we deliberately do not cover. | Open: filed 2026-08-30 with the wave-2 description work; the mapping is a session of reading, not a line edit. |
| AST-B | **The generator-evaluator claim in `agent-evals` matches Anthropic's harness-design article and does not cite it.** Anthropic (read 2026-08-30): "tuning a standalone evaluator to be skeptical is more tractable than making a generator self-critical" — the doctrine `agent-evals` already carries. Add the dated citation where the claim lives, so the next reader can tell doctrine that converged with the field from doctrine invented here. | Open: filed 2026-08-30; deferred to the evals-day wave (AST-05) so the citation lands beside the first executed eval run rather than alone. |
| AG-04a | **This board's own priority is a scalar.** The header computes `P = blast × (1 + age_runs) / effort` while `audit.md` now argues that publishing axes beats multiplying them. The two are not the same claim — the audit is doctrine this pack ships to users about somebody else's system, the board is one repository's own triage — but the tension is real and it is not resolved by saying so. The method's home is the umbrella board (`sshlg-skills/docs/evidence/backlog.md`), which another agent holds; the check added here deliberately scopes to documents under `plugins/`, so it neither polices nor blesses this file. Referred to the umbrella. | Open: the board's formula and `audit.md` disagree, and neither has been chosen. |
16 changes: 16 additions & 0 deletions docs/evidence/verification.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,22 @@ This file exists because its absence read as zero exposure. `sshlg-skills` board

---

## Shipped state — v0.17.0 (2026-08-30)

Measured on the release-candidate tree before the tag exists. Wave-2 of the 2026-08-29
family audit: AST-02/03/04/06/11 plus the harness-engineering externals read 2026-08-30.

| REQ | What ships | How it was confirmed | Confirmed |
|---|---|---|---|
| R-01 | AST-02: the quoted trigger "checker node" lives in exactly one description | `grep -rn '"checker node"' plugins/agent-stack/skills/*/SKILL.md` → one hit, `agent-evals/SKILL.md:12`; `grep -c 'checker before' agent-orchestrator/SKILL.md` → 4, so the orchestrator still states its own claim in prose | **observed** |
| R-02 | AST-11: orchestrator's single-word triggers narrowed | description now carries `"agent system"` and `"агентная система"` (SKILL.md:10,12); neither bare `"agent"` nor bare `"агент"` remains as a quoted trigger there | **observed** |
| R-03 | ReAct advertised as a phrase, not a bare word | `agent-harness/SKILL.md:11` carries `"ReAct loop", "react pattern"`; the only quoted `"ReAct"` trigger is gone, so the umbrella hook can mirror the phrase without firing on the React framework | **observed** |
| R-04 | AST-06: the harness↔orchestrator boundary enumerates its four seams and names the one home | §Boundaries lists tools.md↔§3, system-prompt.md↔§10, static-or-dynamic→graph-engineering.md, context-engineering.md↔system-prompt.md, and states shape-of-the-work doctrine has ONE home (`agent-orchestrator/references/graph-engineering.md`); the intro no longer claims "the shape of the work"; `check_one_home_per_fact` (shingle floor 20) passes on the rewrite | **observed** |
| R-05 | AST-03/04: interop's external claims carry dates | "twelve months before **2026-08-13**" anchors the moved-spec list; the v0.3 and `binds` "still" claims each end "as of 2026-08-13"; the neighbourhood table opens "**Verdicts as of 2026-08-13**" | **observed** |
| R-06 | Harness engineering cited as the outside term, dated, with the deferred work filed | intro cites OpenAI's and Anthropic's articles (both read 2026-08-30) and the ARC-AGI-3 result 13.3%→38.3% at a sixth of the tokens (as reported 2026-08-30); `"harness engineering"` joins the triggers; board rows AST-A1 and AST-B filed in `docs/evidence/backlog.md` | **observed** |
| R-07 | Every description stays inside the working limit after the edits | `python3 test/validate.py` → `OK: agent-stack structurally valid (13 checks = 9 named + 4 per-skill, 4 skill(s), v0.17.0)`; measured 967 / 963 / 970 / 961 chars against the 970 limit; `audit_skill.py --house` → `0 GAP, 14 PASS` for each of the three edited skills | **observed** |
| R-08 | The whole gate is green on the bumped tree | `npm test` → validator OK line above, `PASS: plant_guard — 9 cases`, `PASS: installer — 11 case(s)`, both residue lines "left nothing" | **observed** |

## Shipped state — v0.16.1 (2026-08-29)

Measured on the release-candidate tree before the tag exists. The wave-1.5 rollout:
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@ssheleg/agent-stack",
"version": "0.16.1",
"version": "0.17.0",
"scripts": {
"test": "python3 test/validate.py && python3 test/plant_guard_test.py && node test/installer_test.js"
},
Expand Down
2 changes: 1 addition & 1 deletion plugins/agent-stack/.claude-plugin/plugin.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"name": "agent-stack",
"displayName": "Agent Stack",
"description": "Four skills: agent-orchestrator — tool-calling loops, pipelines with checkpoints, provider routing with fallback, memory architecture, plus the wallet side of reselling LLM access; agent-evals — run/trace/thread evals, LLM judges, and fixtures grown from production; agent-interop — MCP servers and clients, A2A agent cards, the MCP Registry, and gateways; agent-harness — system prompts, tool shaping, workflow-vs-agent, and auditing an agent system.",
"version": "0.16.1",
"version": "0.17.0",
"author": {
"name": "ssheleg",
"url": "https://x.com/sshlg93"
Expand Down
31 changes: 25 additions & 6 deletions plugins/agent-stack/skills/agent-harness/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ description: >-
instead of a score, plus a scanner. Carries Pi as a worked kernel implementation — SDK, RPC
and extension seams — for embedding or extending a harness. Triggers - "system prompt",
"tool description", "agent picks the wrong tool", "agent loops forever", "prompt
engineering", "ReAct", "workflow or agent", "static or dynamic", "audit this agent",
engineering", "harness engineering", "ReAct loop", "react pattern", "workflow or agent",
"static or dynamic", "audit this agent",
"embed an agent", "agent SDK", "Pi harness", "системный промпт", "агент не вызывает тул",
"аудит агента", "встроить агента". Not for the loop's plumbing, its evals, or its
protocols — those are siblings.
Expand All @@ -18,7 +19,13 @@ description: >-

`agent-orchestrator` wires the loop. `agent-evals` proves it behaves. `agent-interop` gets
it talking to other processes. **This skill is the layer between them and the model: the
prompt, the tools, and the shape of the work.**
prompt, the tools, and the choice of technique.** The outside term for this ground is
**harness engineering** — OpenAI's article of that name (`openai.com/index/harness-engineering`,
read 2026-08-30) and Anthropic's harness-design guidance
(`anthropic.com/engineering/harness-design-long-running-apps`, read 2026-08-30) both name
this same layer, and its leverage is measured: on ARC-AGI-3, harness-level changes alone
moved a fixed model from 13.3% to 38.3% while spending a sixth of the tokens (as reported
2026-08-30).

It runs in both directions. Building one and auditing one are the same checklist read
forwards and backwards, which is why they live together here.
Expand Down Expand Up @@ -146,10 +153,22 @@ is then unfalsifiable — including this audit. Say so first, and make it the fi

**Against `agent-orchestrator`.** That skill owns the loop's *plumbing*: iteration guards,
trimming, sub-agent dispatch, provider routing, memory layers, checkpoints. This one owns
what the model is *told*. They meet in one place:
`agent-orchestrator/references/context-engineering.md` covers **compaction** — what to drop
when the window fills — while this skill's `system-prompt.md` covers what to put there in
the first place. Filling and emptying, two files.
what the model is *told*. They meet at four seams, each crossing in exactly one place:

1. **Describing a tool** so the model picks the right one is this skill's
`references/tools.md`; assembling the tool *list* per request from capability flags is
the orchestrator's §3, which points here for the wording.
2. **What the prompt says** — altitude, vocabulary, enumerated statuses — is this skill's
`references/system-prompt.md`; *rebuilding* that prompt per request, in the same pass
as the tools, is the orchestrator's §10.
3. **The shape of the work** — the static/dynamic table, the fake-edge test, the checker
before a convergence — has ONE home, and it is not here:
`agent-orchestrator/references/graph-engineering.md`. This skill's static-or-dynamic
section stops at the decision and links there for the model.
4. **The context window:** `agent-orchestrator/references/context-engineering.md` covers
**compaction** — what to drop when the window fills — while this skill's
`system-prompt.md` covers what to put there in the first place. Filling and emptying,
two files.

**Against `agent-evals`.** That skill measures whether an agent behaves, from execution
records. This one reviews how it was *built*, from its source and prompts. An audit that
Expand Down
15 changes: 9 additions & 6 deletions plugins/agent-stack/skills/agent-interop/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,19 +24,20 @@ registry, and the gateway between them.

## Rule zero — a protocol claim without a date is a guess

Every one of these specifications moved in the last twelve months, and each moved in a way
that silently breaks code written against the previous revision. Concretely, and each one
measured against the live specification rather than recalled:
Every one of these specifications moved in the twelve months before **2026-08-13** — the
date every reference below was read — and each moved in a way that silently breaks code
written against the previous revision. Concretely, and each one measured on that date
against the live specification rather than recalled:

- MCP replaced the `initialize` handshake with `server/discover` and made the protocol
**stateless**; a client that opens with `initialize` is speaking a revision that is on its
way out.
- MCP **deprecated `sampling`, `roots` and `logging`** — three things an older model will
reach for first, because for a year they were the interesting part of the client side.
- A2A renamed its wire surface between v0.x and v1.0, and SDKs and blog posts still document
v0.3.
- A2A renamed its wire surface between v0.x and v1.0, and SDKs and blog posts still
documented v0.3 as of 2026-08-13.
- agentgateway **deprecated `binds`** in favour of `gateways` — while its own overview page
still introduces `binds` as a core concept.
still introduced `binds` as a core concept as of 2026-08-13.

So: **every reference in this skill opens with a `**Spec pinned:**` line**, and
`test/validate.py` fails the build without one. That is a mechanical check, not an
Expand Down Expand Up @@ -119,6 +120,8 @@ it); prompt wording; a single LLM call in a script.
## The neighbourhood — one verdict each

Named so an agent stops guessing, with the verdict stated rather than implied.
**Verdicts as of 2026-08-13**, the day this skill's references were pinned — this ground
moves, so re-verify anything marked *watch* before building on it.

| Thing | What it actually is | Verdict |
|---|---|---|
Expand Down
4 changes: 2 additions & 2 deletions plugins/agent-stack/skills/agent-orchestrator/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@ description: >-
architecture, retrieval and decay, context budgets, sub-agent coordination, error hierarchies; the
work as a graph — parallel layers, fake edges, a checker before convergence; for resale:
tiered wallets, one markup boundary, two-phase commit across database and provider API,
spend-delta polling, budget and loop guards, per-tenant keys. Triggers - "agent",
spend-delta polling, budget and loop guards, per-tenant keys. Triggers - "agent system",
"orchestrator", "tool calling", "sub-agent", "LLM router", "fallback chain", "human in the
loop", "memory layer", "LLM billing", "token wallet", "checker node", "агент", "оркестратор",
loop", "memory layer", "LLM billing", "token wallet", "агентная система", "оркестратор",
"суб-агент", "роутер моделей", "человек в цикле", "слой памяти", "биллинг LLM", "граф задач".
Not for a single LLM call in a script, or prompt wording.
---
Expand Down
Loading