diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index db1131f..8f6fa5d 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -1,4 +1,5 @@ { + "$schema": "https://json.schemastore.org/claude-code-marketplace.json", "name": "agent-stack", "owner": { "name": "ssheleg", @@ -11,7 +12,7 @@ "displayName": "Agent Stack", "source": "./plugins/agent-stack", "description": "Four skills: agent-orchestrator — tool-calling loops, pipelines with checkpoints, provider routing with fallback, memory architecture, plus the wallet side of reselling LLM access; agent-evals — run/trace/thread evals, LLM judges, and fixtures grown from production; agent-interop — MCP servers and clients, A2A agent cards, the MCP Registry, and gateways; agent-harness — system prompts, tool shaping, workflow-vs-agent, and auditing an agent system.", - "version": "0.17.0", + "version": "0.17.1", "author": { "name": "ssheleg", "url": "https://x.com/sshlg93" diff --git a/CHANGELOG.md b/CHANGELOG.md index f143aac..6a32431 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,46 @@ # Changelog +## v0.17.1 — the evals run for the first time, and the tails go to zero + +Wave-3 of the 2026-08-29 family audit (rows AST-05, AST-07, AST-08, AST-09, AST-10), +plus the two board rows wave 2 filed (AST-A1, AST-B). All fixes and docs — no +description or trigger changes, hence a patch. + +- **AST-05: the eval suite is executed for the first time.** `test/evals/RESULTS.md` + gains two dated rows — haiku and sonnet, 2026-08-31 — each a fresh blind subagent + per trigger query against the family's 28 skill descriptions, plus all three + scenarios scored line by line. Both models: 11/12 triggers. Scenario lines: 10/12 + (haiku), 9/12 (sonnet). Each miss is named — haiku false-triggered `agent-harness` + on "rewrite this one system prompt so it sounds friendlier" (q09), sonnet answered + `none` on the Russian harness-audit query (q04) — and the Method section states the + protocol and its three limits rather than presenting the rates as clean-room. +- **AST-B: the generator-evaluator citation lands where the doctrine lives.** + `agent-evals` §5a now cites Anthropic's *tuning a standalone evaluator to be + skeptical is more tractable than making a generator self-critical* (read + 2026-08-30) beside the sentence making the same claim, marked as convergence + rather than invention — beside the first executed eval run, as the board row asked. +- **AST-A1: the nine-subsystem coverage check is done and the row closes.** The ninth + subsystem the filing row could not name is **Automation**; all nine map to existing + doctrine with a `file:line` each, and the suspected identity/approval-policy gap + resolves as a split verdict — approval policy covered (`governance.md`), + authentication mechanics a named delegation (`layers.md:78-80`), not a hole. The + map lives in the board row (`docs/evidence/backlog.md`), not in a new reference. +- **AST-07: the README stops counting its references.** "Twenty references" had + drifted to 24 actual files within a week of being written; the aggregate count is + dropped in favour of the per-skill counts that are still true, with the reason + stated in place. +- **AST-10: the orchestrator's reference index lists all 11 references.** The three + memory rows — `memory-architecture.md`, `memory-lifecycle.md`, + `memory-landscape.md` — join the index table, each paraphrasing its own + "Load this when" line. +- **AST-08: `license: MIT` in all four skill front matters** (was 1 of 4 — + `agent-evals` alone carried it). Every front matter re-checked with + `yaml.safe_load` after the edit. +- **AST-09: `$schema` in both manifests** — `claude-code-plugin-manifest.json` for + `plugin.json`, `claude-code-marketplace.json` for `marketplace.json`, the two + schemastore addresses that resolve (the third candidate, `claude-code-plugin.json`, + is a 404 and two siblings currently point at it; referred, not copied). + ## v0.17.0 — the descriptions stop colliding, and the claims get their dates Wave-2 of the 2026-08-29 family audit (rows AST-02, AST-03, AST-04, AST-06, AST-11), plus diff --git a/README.md b/README.md index 7e9e05c..d97c2cf 100644 --- a/README.md +++ b/README.md @@ -33,8 +33,9 @@ Part of the [ssheleg skill family](https://github.com/ssheleg/sshlg-skills). Four skills — `agent-orchestrator` for wiring the loop, `agent-evals` for proving it behaves, `agent-interop` for everything it talks to outside its own process, -`agent-harness` for what it is **told** — and twenty references they load on demand, -plus one scanner. +`agent-harness` for what it is **told** — and the references they load on demand, +plus one scanner. The per-skill sections below carry their own counts; no total is +stated here, because the last one drifted stale within a week of being written. **The orchestrator** (`SKILL.md`) — what the agent reads first: diff --git a/docs/evidence/backlog.md b/docs/evidence/backlog.md index 3050258..1606463 100644 --- a/docs/evidence/backlog.md +++ b/docs/evidence/backlog.md @@ -26,6 +26,8 @@ row per line, appended, so two runs closing two rows do not land on the same lin | AG-04 + AG-05 | **The audit document refused a score and computed one.** `agent-harness/references/audit.md:20-25` said *"A prioritized change plan … **Not a score**"* and *"pass/fail with a named failure condition beats a scalar that names no fix"*; `:114` then computed `P = blast × confidence / effort` and `:128` ordered the plan by it. The manifesto backs the refusal (`pod-manifesto/manifesto.md:424`, *"these axes are not a fake numerical score"*) and names FOUR axes at `:419-422` — Impact, **Irreversibility**, Uncertainty, **Coordination** — of which two appeared nowhere in the pack (`grep -ci irreversib` → **0**, `grep -ci coordinat` → **0**) while `effort`, a **cost**, had been substituted into their place. So the file held neither position. | 2026-08-18 manifesto-conformance audit, rows AG-04 and AG-05 | 3 | 2 | 2 | **3.0** | **closed 2026-08-20** — the position taken is the one the document already argued for: publish the axes, drop the arithmetic. §*Priority — four axes, and no scalar* declares `` and states each with its High·Medium·Low values, cited to `manifesto.md:419-422`. The ordering rule is **the first axis that separates two findings**, which is inspectable in the direction the formula was not: `3 × 1 / 3` and `1 × 1 / 1` both print **1**, so *harms a user, judgement-tier, its own project* and *annoys a maintainer, judgement-tier, an hour* arrived at one number and the inputs could not be recovered from it. `effort` keeps its job — sizing the plan's first three items — and loses its rank. `check_priority_axes_are_the_manifesto_s_and_carry_no_scalar` enforces the axis list as an **equality** (a fifth axis nobody sourced is as wrong as a missing one), each axis in the prose, the spelled count, both restored axes present somewhere in the pack, no prescribed scalar anywhere in it, and no plan ordered by one. Three negative self-tests in `validate.yml`, each watched at **exit 1**: `effort` substituted back for an axis, the formula reinstated as a standalone line, and `ordered by P`. A fourth plant run by hand — both restored axes renamed away across the pack — also refused. **One false positive was found by watching the check refuse a correct document**: the first draft matched the formula anywhere on a line, including the paragraph that *records its removal*. The guard now reads a whole-line prescription and not a citation, which is the umbrella's own rule — name a dead command, never claim it. | | AG-06 | **`test/plant_guard_test.py:34` leaked 8 nameless `tmpXXXXXXXX` trees per run and the gate said nothing.** Measured 2026-08-20: `find "$TMPDIR" -maxdepth 4 -type f -path '*/copy/sub/b.sh' \| wc -l` → **2568 before one run, 2576 after**. The fixture is byte-identical (md5 `623a086d10a04940573c31cbebb93e31`) in this repository and `seo-aeo-audit`. | AG-06, 2026-08-20 | 2 | 1 | 1 | **4.0** | **closed 2026-08-20 by PORTING, not reinventing** — `test/residue.py` comes from `make-skill`, which measured the same defect first and shipped the answer; a second implementation of one ledger is how two gates disagree about what a clean run leaves. Differences: the `agent-stack-test-` prefix, the suite it accounts for, and its opening paragraph. Every tree goes through `residue.workspace()`, a failing case KEEPS its tree (a plant is debugged by reading the copy it landed in), and both gate commands print a residue line — `npm test` now ends `residue: this run left nothing — 8 temp tree(s) created, 8 removed`, and `test/validate.py` carries the same tolerant import so the next thing that writes to `$TMPDIR` shows up in its own output. **The port alone was not enough, and watching it fail is what showed it:** the leak re-planted as a bare `tempfile.mkdtemp()` still PASSED and the ledger printed *"left nothing — 0 temp tree(s) created"* — a bypass reporting clean is indistinguishable from no leak. `check_temp_trees_go_through_the_residue_ledger` closes that: no `test/*.py` but `residue.py` may call `mkdtemp` or `TemporaryDirectory`, read from the source with comments and docstrings stripped so an explanation of the ban is not read as a violation. Both directions watched at **exit 1**, and one negative self-test each in `validate.yml`. | | AG-07 + AG-08 | **The ledger asserted a state that was false at HEAD.** Three sections of `docs/evidence/verification.md` were headed `(AG-0N, unreleased)` and carried *"No release: the version stays 0.11.1 and the CHANGELOG is untouched, so every row below is measured on the working tree rather than on a published artifact"* — while **v0.12.0 was tagged, on npm and in the CHANGELOG**. Under those paragraphs **35 rows** (15 + 11 + 9) all read `**verified**`, against the file's own opening at `:4-5`: *"A row sits at `never` until somebody has watched its check pass on the shipped artifact — not on a branch, not in a plan."* | 2026-08-18 manifesto-conformance audit, rows AG-07 and AG-08 | 3 | 1 | 1 | **6.0** | **closed 2026-08-20** — the rows are not re-graded on a promise: the gate was **re-run on the shipped artifact**. `git archive v0.12.0` (tag `7937c35`, released 2026-08-19T19:30:00Z, `npm view @ssheleg/agent-stack version` → `0.12.0`) into a clean tree, then `python3 test/validate.py` → `OK: agent-stack structurally valid (13 checks, 4 skill(s), v0.12.0)` exit **0**, and `python3 test/plant_guard_test.py` → `PASS: plant_guard — 9 cases` exit **0**. All three paragraphs now say what shipped and carry that receipt, and each records what it used to say. `check_the_ledger_matches_what_shipped` enforces both directions: a section declaring no release may carry **no** `verified` row, a section whose *"version stays X"* is behind the CHANGELOG's newest release is refused, and a section claiming `shipped in vX` needs a `## vX` in the CHANGELOG. Both plants watched at **exit 1**, the first naming the shipped defect exactly — *"declares no release and carries 15 row(s) marked **verified**"* — with a negative self-test each in `validate.yml`. | +| AST-A1 | **Coverage check against OpenAI's nine-subsystem harness taxonomy.** Filed 2026-08-30 with the wave-2 description work; the suspected gap was identity/approval-policy. | 2026-08-30 harness-articles digest | 2 | 1 | 2 | **2.0** | **closed 2026-08-31** — the ninth subsystem the filing row could not name is **Automation**: the taxonomy as recovered via the `kenhuangus.substack.com` retelling (re-read 2026-08-31; `openai.com/index/harness-engineering` still 403) is Automation, Identity/Intent, Memory, Observability, Runtime Control, Evaluation, Scalability, Token Budget, Extensions. The map, one verdict per subsystem, each address checked at this commit: **Automation** (persistent shell, file-patch primitives) — covered as the worked kernel, `agent-harness/references/pi.md:33` (`read/write/edit/bash` and a loop) and `:191-198` (OpenShell, where execution actually runs), with the agent–computer interface in `agent-harness/references/tools.md`; **Identity/Intent** — split verdict, and it resolves the suspected gap: *approval policy* is covered — `agent-orchestrator/references/governance.md:52-56` (a deterministic limit or a human, never a classifier's confidence), `:24-37` (four boundaries with per-tool authorisation), `:75-91` (the audit row: who ran it, who changed the policy, which policy version) — while *authentication mechanics* (PKCE, workload federation) are a named delegation rather than a hole: `agent-harness/references/layers.md:78-80` (*identity and secrets — a credential store, never the loop*), with the MCP OAuth surface at `agent-interop/references/mcp.md:201-210`; **Memory** — `agent-orchestrator/references/memory-architecture.md`, `memory-lifecycle.md`, `memory-landscape.md`, plus `agent-orchestrator/SKILL.md:225` (§7); **Observability** — `agent-evals/SKILL.md:48-50` (traces durable and queryable as the precondition), `:273` (§7 what to instrument first), `agent-orchestrator/SKILL.md:264` (§9); **Runtime Control** — `agent-orchestrator/references/governance.md:127-138` (a sandbox protects the host; egress allowlists; credentials never enter) and `agent-harness/references/layers.md:59-77` (why the boundary is the environment's, and how deciding the layer decides the audit); **Evaluation** — `agent-evals/SKILL.md` end to end, the in-loop half at §5a (`:164`), the audit half `agent-harness/references/audit.md`; **Scalability** — `agent-orchestrator/references/runtime.md` (thread lifecycles: checkpoint every iteration, interrupt/resume, double-texting, schedules), `references/graph-engineering.md` (parallel layers and what a host executes on fan-out), `references/provider-lifecycle.md` (workspace lifecycle, fleet budgets); **Token Budget** — `agent-orchestrator/references/context-engineering.md` (compaction ladder, token estimation, circuit breaker), the money meter in `references/llm-proxy-billing.md`; **Extensions** — `agent-harness/references/pi-sdk.md:247` (the extension seams), the outward protocol in `agent-interop/references/mcp.md`, and skill construction deliberately NOT covered — that is `make-skill`, the boundary both descriptions already draw. Verdict: **nine of nine either covered with an address or delegated with the delegation named; no reference gains a rewrite from this row.** The map lives here per this board's convention — the row is the record, and a doctrine home for it would be a tenth document pointing at nine existing ones. | +| AST-B | **The generator-evaluator claim in `agent-evals` matched Anthropic's harness-design article and did not cite it.** | 2026-08-30 harness-articles digest | 3 | 1 | 1 | **6.0** | **closed 2026-08-31** — the citation lands where the doctrine lives, `agent-evals/SKILL.md` §5a: Anthropic's *tuning a standalone evaluator to be skeptical is more tractable than making a generator self-critical* (`anthropic.com/engineering/harness-design-long-running-apps`, read 2026-08-30) now sits beside the sentence that makes the same claim — the convergence depends on the checker, not on the branches grading themselves — dated, and marked as convergence rather than invention. Landed beside the first executed eval run (AST-05, `test/evals/RESULTS.md`), as the filing row asked. `check_one_home_per_fact` stays green on the quoted sentence: a citation, not a restatement. | | AG-09 | **Three of four skill descriptions were past the house working limit**, one with **five** characters of headroom before the hard 1024 cap: `agent-orchestrator/SKILL.md:3` at **1019**, `agent-evals` **986**, `agent-interop` **983** (measured by `make-skill/scripts/audit_skill.py --house`, whose `DESC_TARGET` is 970). A description at 99% of cap cannot absorb the *and not for X* clause the next near-miss neighbour requires — which is the clause that stops a sibling stealing its triggers. | AG-09, 2026-08-20 | 2 | 1 | 1 | **4.0** | **closed 2026-08-20** — prose compressed, **no trigger phrase touched**: quoted-trigger counts before and after are 19→19, 12→12 and 14→14, `lost: []` in each case. Now 964 / 963 / 970 against the 970 limit, and `audit_skill.py --house` reports `0 GAP` for all four skills (`agent-harness` was already at 916). The limit is also a **gate** now, not only an external auditor's opinion: `test/validate.py` refuses anything past `DESC_WORKING_LIMIT = 970`, sourced by name to make-skill's `DESC_TARGET` rather than re-decided here. Watched at **exit 1** on a description padded to 1014 chars — under the hard cap, over the working limit, which is the band the old check could not see — plus a negative self-test in `validate.yml`. | ## Open, and why @@ -34,6 +36,4 @@ row per line, appended, so two runs closing two rows do not land on the same lin |---|---|---| | AG-06a | **A temp tree created outside `test/` is still invisible.** `check_temp_trees_go_through_the_residue_ledger` reads this repository's own suite. A tree made by anything else — a script under `plugins/`, a future helper outside `test/` — is not accounted for, and `residue.report()` counts only what `workspace()` handed out. The mechanism catches a *reviewed* bypass; it does not observe `$TMPDIR`. Observing it would mean a run claiming ownership of directories it cannot prove it made, which is the reaping mistake `agent-sync` refuses by design. | Open: the ledger reads this repository's own suite, so a tree made anywhere else is still unattributable. | | AG-06b | **The 2592 existing trees are reported and left alone.** `find "$TMPDIR" -maxdepth 4 -type f -path '*/copy/sub/b.sh' \\| wc -l` → **2592** on 2026-08-20, of which **16 were added by this run's own verification** — one run of the leaking fixture before the port, and one of the v0.12.0 archive's copy of it. They are plain `tmpXXXXXXXX` and indistinguishable from every other program's, so they are named rather than swept. The identical fixture still leaks in **`seo-aeo-audit`**, which another agent holds; referred, not edited. | Open by decision: state this run cannot prove is its own is reported, never deleted. | -| AST-A1 | **Coverage check against OpenAI's nine-subsystem harness taxonomy.** OpenAI's harness-engineering article (`openai.com/index/harness-engineering`, read 2026-08-30) enumerates nine subsystems a harness owns — identity/intent, memory, observability, runtime control, evaluation, scalability, token budget, extensions among them. `agent-harness` now cites the term (v0.17.0) but nobody has walked our references against that list; the suspected gap is **identity / approval policy**, which `references/layers.md` currently waves at as "usually somebody else's job". The check is a mapping table with a `file:line` per subsystem, or a named refusal per subsystem we deliberately do not cover. | Open: filed 2026-08-30 with the wave-2 description work; the mapping is a session of reading, not a line edit. | -| AST-B | **The generator-evaluator claim in `agent-evals` matches Anthropic's harness-design article and does not cite it.** Anthropic (read 2026-08-30): "tuning a standalone evaluator to be skeptical is more tractable than making a generator self-critical" — the doctrine `agent-evals` already carries. Add the dated citation where the claim lives, so the next reader can tell doctrine that converged with the field from doctrine invented here. | Open: filed 2026-08-30; deferred to the evals-day wave (AST-05) so the citation lands beside the first executed eval run rather than alone. | | AG-04a | **This board's own priority is a scalar.** The header computes `P = blast × (1 + age_runs) / effort` while `audit.md` now argues that publishing axes beats multiplying them. The two are not the same claim — the audit is doctrine this pack ships to users about somebody else's system, the board is one repository's own triage — but the tension is real and it is not resolved by saying so. The method's home is the umbrella board (`sshlg-skills/docs/evidence/backlog.md`), which another agent holds; the check added here deliberately scopes to documents under `plugins/`, so it neither polices nor blesses this file. Referred to the umbrella. | Open: the board's formula and `audit.md` disagree, and neither has been chosen. | diff --git a/docs/evidence/verification.md b/docs/evidence/verification.md index 67a9c14..2193d82 100644 --- a/docs/evidence/verification.md +++ b/docs/evidence/verification.md @@ -10,6 +10,22 @@ This file exists because its absence read as zero exposure. `sshlg-skills` board --- +## Shipped state — v0.17.1 (2026-08-31) + +Measured on the release-candidate tree before the tag exists. Wave-3 of the 2026-08-29 +family audit: AST-05/07/08/09/10 plus board rows AST-A1 and AST-B closed. + +| REQ | What ships | How it was confirmed | Confirmed | +|---|---|---|---| +| R-01 | AST-05: two dated eval rows exist, with a Method line | `test/evals/RESULTS.md` carries rows dated 2026-08-31 for haiku and sonnet — 24 fresh blind trigger probes (one subagent per query per model), 6 scenario runs scored line by line; misses named (q09 haiku false trigger, q04 sonnet `none`); `python3 test/evals_validate.py` → `OK: 12 trigger cases and 3 scenarios validate`, `--self-test` → planted invalid class caught | **observed** | +| R-02 | AST-B: the Anthropic generator-evaluator citation sits in `agent-evals` §5a, dated read 2026-08-30 | `grep -n 'standalone evaluator' plugins/agent-stack/skills/agent-evals/SKILL.md` → hit inside §5a beside "the convergence depends on **it** rather than on the branches"; `check_one_home_per_fact` (shingle floor 20) green on the quoted sentence | **observed** | +| R-03 | AST-A1: nine subsystems mapped, row closed with the map | ninth subsystem recovered as **Automation** (kenhuangus.substack.com retelling re-read 2026-08-31; `openai.com/index/harness-engineering` → 403); every `file:line` in the map re-resolved on the edited tree (line numbers shift with this release's own edits, so they were re-measured after, not before); the suspected identity/approval-policy gap closed as covered-plus-named-delegation | **observed** | +| R-04 | AST-07: no aggregate reference count in the README | `grep -c 'twenty references' README.md` → 0; actual count at this commit: `ls plugins/agent-stack/skills/*/references/*.md \| wc -l` → 24 (11 orchestrator + 7 harness + 6 interop), which is why the aggregate was dropped rather than corrected — it moved once already | **observed** | +| R-05 | AST-10: the orchestrator index lists 11 of 11 | `grep -c 'references/' agent-orchestrator/SKILL.md` index table rows → 11; both-directions check in `test/validate.py` green (no orphan, no dangling) | **observed** | +| R-06 | AST-08: `license: MIT` in 4 of 4 front matters, YAML-safe | `yaml.safe_load` over each front matter → keys `[description, license, name]`, descriptions intact at 967/963/970/961 chars, all ≤ the 970 working limit | **observed** | +| R-07 | AST-09: `$schema` present and resolving in both manifests | `curl -sL -o /dev/null -w '%{http_code}'` → `claude-code-plugin-manifest.json` 200, `claude-code-marketplace.json` 200 (the third candidate `claude-code-plugin.json` → 404, not used); `claude plugin validate . --strict` and `plugins/agent-stack --strict` → both `✔ Validation passed` | **observed** | +| R-08 | The whole gate is green on the bumped tree | `npm test` → `OK: agent-stack structurally valid (13 checks = 9 named + 4 per-skill, 4 skill(s), v0.17.1)`, `PASS: plant_guard — 9 cases`, `PASS: installer — 11 case(s)`, residue lines "left nothing" | **observed** | + ## Shipped state — v0.17.0 (2026-08-30) Measured on the release-candidate tree before the tag exists. Wave-2 of the 2026-08-29 diff --git a/package.json b/package.json index b111cab..53a5e4c 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@ssheleg/agent-stack", - "version": "0.17.0", + "version": "0.17.1", "scripts": { "test": "python3 test/validate.py && python3 test/plant_guard_test.py && node test/installer_test.js" }, diff --git a/plugins/agent-stack/.claude-plugin/plugin.json b/plugins/agent-stack/.claude-plugin/plugin.json index 115336f..32f751c 100644 --- a/plugins/agent-stack/.claude-plugin/plugin.json +++ b/plugins/agent-stack/.claude-plugin/plugin.json @@ -1,8 +1,9 @@ { + "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json", "name": "agent-stack", "displayName": "Agent Stack", "description": "Four skills: agent-orchestrator — tool-calling loops, pipelines with checkpoints, provider routing with fallback, memory architecture, plus the wallet side of reselling LLM access; agent-evals — run/trace/thread evals, LLM judges, and fixtures grown from production; agent-interop — MCP servers and clients, A2A agent cards, the MCP Registry, and gateways; agent-harness — system prompts, tool shaping, workflow-vs-agent, and auditing an agent system.", - "version": "0.17.0", + "version": "0.17.1", "author": { "name": "ssheleg", "url": "https://x.com/sshlg93" diff --git a/plugins/agent-stack/skills/agent-evals/SKILL.md b/plugins/agent-stack/skills/agent-evals/SKILL.md index 41fc1dd..b274da2 100644 --- a/plugins/agent-stack/skills/agent-evals/SKILL.md +++ b/plugins/agent-stack/skills/agent-evals/SKILL.md @@ -168,6 +168,13 @@ the one most systems are missing: a **checker node** sitting between a parallel the node that consumes it. Its only job is *usable / not usable*, and the convergence depends on **it** rather than on the branches — otherwise the gate has a bypass. +**That split is measured practice, not only this pack's position.** Anthropic's +harness-design guidance (`anthropic.com/engineering/harness-design-long-running-apps`, +read 2026-08-30) reports that *tuning a standalone evaluator to be skeptical is more +tractable than making a generator self-critical* — the same reason the verdict belongs to +a separate node rather than to the branches grading themselves. Dated and cited so a +reader can tell doctrine that converged with the field from doctrine invented here. + It matters here because it is the same machinery as §5, positioned differently: diff --git a/plugins/agent-stack/skills/agent-harness/SKILL.md b/plugins/agent-stack/skills/agent-harness/SKILL.md index 020ac66..59a5667 100644 --- a/plugins/agent-stack/skills/agent-harness/SKILL.md +++ b/plugins/agent-stack/skills/agent-harness/SKILL.md @@ -13,6 +13,7 @@ description: >- "embed an agent", "agent SDK", "Pi harness", "системный промпт", "агент не вызывает тул", "аудит агента", "встроить агента". Not for the loop's plumbing, its evals, or its protocols — those are siblings. +license: MIT --- # Agent harness — what the agent is told, and how to audit what someone else told theirs diff --git a/plugins/agent-stack/skills/agent-interop/SKILL.md b/plugins/agent-stack/skills/agent-interop/SKILL.md index 6ce0cc0..1dcbf95 100644 --- a/plugins/agent-stack/skills/agent-interop/SKILL.md +++ b/plugins/agent-stack/skills/agent-interop/SKILL.md @@ -12,6 +12,7 @@ description: >- "agentgateway", "tool federation", "MCP-сервер", "карточка агента", "интероперабельность агентов", "реестр MCP", "шлюз для агентов". Not for designing one server's tool set, nor for a skill's own construction — that is make-skill. +license: MIT --- # Agent interop — the protocols an agent speaks outside its own process diff --git a/plugins/agent-stack/skills/agent-orchestrator/SKILL.md b/plugins/agent-stack/skills/agent-orchestrator/SKILL.md index b4c6db1..adf5273 100644 --- a/plugins/agent-stack/skills/agent-orchestrator/SKILL.md +++ b/plugins/agent-stack/skills/agent-orchestrator/SKILL.md @@ -12,6 +12,7 @@ description: >- loop", "memory layer", "LLM billing", "token wallet", "агентная система", "оркестратор", "суб-агент", "роутер моделей", "человек в цикле", "слой памяти", "биллинг LLM", "граф задач". Not for a single LLM call in a script, or prompt wording. +license: MIT --- # Agent Orchestrator — Production Best Practices @@ -245,9 +246,7 @@ answer built from nothing, in no error log. **Design a memory layer from [`references/memory-architecture.md`](references/memory-architecture.md)**, not from this table — it also carries the context-budget trap, layer 0 carryover and workspace scale. -The write path is [`references/memory-lifecycle.md`](references/memory-lifecycle.md); -what to build on and measure with is -[`references/memory-landscape.md`](references/memory-landscape.md). +The write path and the build-or-adopt question have their own rows in the index below. ## 8. Self-Learning Feedback Loops Three cycles feed the memory layers, and they differ by what supplies the signal: a failed @@ -373,15 +372,18 @@ heading — the ones that were learned by getting them wrong: ## References Each file opens with its own **Load this when** line — the authoritative trigger lives -there, so this table stays an index and the two cannot drift apart. +there; this table is only an index. | File | Read it when | |---|---| -| [`references/graph-engineering.md`](references/graph-engineering.md) | you are deciding the **shape of the work** — the fake-edge test, the diamond, the checker node, static versus dynamic, and what the host actually runs | +| [`references/graph-engineering.md`](references/graph-engineering.md) | you are deciding the **shape of the work** — fake edges, the diamond, the checker node, static versus dynamic | | [`references/pipeline.md`](references/pipeline.md) | one loop is **not the shape** — the planned path, its checkpoints, resume, and the interrupt that asks a person | | [`references/patterns.md`](references/patterns.md) | you need the **data models and algorithms** under the body | +| [`references/memory-architecture.md`](references/memory-architecture.md) | an agent is **being given memory**, or memory is misbehaving | +| [`references/memory-lifecycle.md`](references/memory-lifecycle.md) | the memory **write path** — what enters, and what leaves | +| [`references/memory-landscape.md`](references/memory-landscape.md) | **build or adopt** a memory layer, and what is settled practice | | [`references/context-engineering.md`](references/context-engineering.md) | the loop is **running out of window** | | [`references/runtime.md`](references/runtime.md) | the agent must **survive a crash, a pause, a second message or a schedule** | | [`references/governance.md`](references/governance.md) | the question is **permission, not cost** — what it may do, and how you prove it | | [`references/llm-proxy-billing.md`](references/llm-proxy-billing.md) | the product **resells LLM access** | -| [`references/provider-lifecycle.md`](references/provider-lifecycle.md) | the question is the **workforce, not the loop** — where providers come from, produced-once/bound-many, knowledge packs, canary trust, workspace lifecycle, fleet budgets | +| [`references/provider-lifecycle.md`](references/provider-lifecycle.md) | the question is the **workforce, not the loop** — provider lifecycle, canary trust, fleet budgets | diff --git a/test/evals/RESULTS.md b/test/evals/RESULTS.md index 94357c0..2436a31 100644 --- a/test/evals/RESULTS.md +++ b/test/evals/RESULTS.md @@ -1,12 +1,46 @@ # Evaluation results -**Status: authored, schema-validated, never executed against a model.** - -CI proves that the files are shaped correctly and that the validator catches a -planted invalid trigger class. It does not run an agent session. Until a dated -row exists below, no behavioral pass rate is claimed. +**Status: executed — first dated rows below (2026-08-31).** CI still proves only +that the files are shaped correctly; the rows are where behavioural claims live, +and each carries the method that produced it. | Date | Version | Model | Trigger pass rate (train / validation) | Scenario lines passed | Installed alongside | Notes | |---|---|---|---|---|---|---| -| — | — | — | — | — | — | no run yet | +| — | — | — | — | — | — | superseded: "no run yet" row retired by the rows below | +| 2026-08-31 | 0.17.1 (tree at release candidate) | claude-haiku (Agent-tool `haiku`) | 5/6 train, 6/6 validation (11/12 overall) | s01 4/4 · s02 2/4 · s03 4/4 (10/12 lines) | ssheleg family: make-skill, task-pipeline (+evidence-docs, project-audit), super-ux (brand-voice, copywriting, ux-audit, ux-flows, ux-foundation, ux-scenarios, vision), sheleg-design, seo-aeo-audit, agent-sync, sheleg-dev (7 skills), telegram-dev (3 skills), agent-stack (4 skills) — 28 descriptions total | Miss: q09 ("Rewrite this one system prompt so it sounds friendlier") false-triggered `agent-harness`. All six positives hit the intended skill; q07/q08/q10/q11 routed to the correct sibling pack (task-pipeline, agent-sync, stripe-billing, task-pipeline), q12 answered `none`. | +| 2026-08-31 | 0.17.1 (tree at release candidate) | claude-sonnet (Agent-tool `sonnet`) | 6/6 train, 5/6 validation (11/12 overall) | s01 3/4 · s02 2/4 · s03 4/4 (9/12 lines) | same 28-skill family list as the haiku row | Miss: q04 ("проверь системный промпт и описания инструментов нашего агента") answered `none` instead of `agent-harness`. All other positives hit; negatives routed to the correct sibling or `none`. q07 was probed twice (a launch retry) — both runs answered task-pipeline. | + +## Method (2026-08-31 rows) + +Wave-3 eval protocol, executed from a Claude Code harness rather than a chat UI: + +- **Triggers:** one FRESH general-purpose subagent per query per model (Agent + tool, `model: haiku` / `model: sonnet`), no shared context between probes. The + probe prompt was the query verbatim plus an instruction to read a file holding + the family's 28 skill names-with-descriptions (built from the members' + `SKILL.md` front matters) and answer with one skill name or `none`. Scoring: a + positive passes only on the intended skill; a negative passes on `none` or any + non-`agent-stack` skill. +- **Known limits, stated rather than hidden:** (1) each subagent's own system + prompt also lists the machine's full installed-skill roster (family plus + foreign packs), so the probe context is the real machine, not a clean room — + sonnet probes mostly answered from that roster without opening the file + (0 tool uses), haiku probes read the file first (1 tool use); answers in + `plugin:skill` form were normalised to the skill name. (2) One probe per query + per model, not the three repetitions `test/evals/README.md` asks for — cost + bound; treat single-probe rates as coarse. (3) The suite names no models, so + haiku and sonnet were chosen; opus was not probed. +- **Scenarios:** per scenario and model, one fresh subagent was given the + scenario query and told to read the named skill file(s) first (simulating the + skill being loaded), then answer. Each `expected_behavior` line was scored + pass/fail by the coordinating agent reading the full answer — an LLM-judge + grade with n=1, not a calibrated judge. A full interactive session (real skill + auto-loading, multi-turn) is not reproducible from this harness; trigger rows + above are the loading evidence, scenario rows are content evidence. +- **What the scenario failures name, so the next edit knows where to aim:** the + one line that failed on BOTH models is s02's *"Records model, prompt and tool + versions with results"* — neither answer surfaced version-recording, though + `agent-evals` §7 states it; s02's judge-calibration line failed on sonnet and + its adversarial-cases line on haiku; s01's *"Names provider failure and + context-pressure behavior"* failed on sonnet. s03 passed 4/4 on both models.