feat(skills): add harness skill for AI-executable validation

[isolomatov-gd]

Closes #147.

What

Adds harness — a model-invocable skill that designs the executable apparatus a coding agent needs to actually run, observe, and validate what it changed, rather than stopping at "compiles and unit tests pass."

SKILL.md is a router, not a builder. Nine steps: classify the gap → discover and extend what the target repo already has → propose placement in its own conventions → load the matching asset → write the specification to FEATURE PLAN folder → HITL gate → build → prove by execution → record a ## Harness inventory in ARCHITECTURE.md.

Five kinds

Independent, often combined. Each answers a different "we cannot verify this."

Kind	Gap	Asset
ACTIONS	cannot act on or observe a running service	cli-mcp-scripts.md
ENVIRONMENT	cannot run it locally at all	devcontainers.md
PROMPTING	cannot author or prove a coding-agent skill or subagent	prompting.md, prompting-writing.md, prompting-proof.md
HOOKS	behavior must hold every time regardless of the model	hooks.md, references/hooks/
AUTOMATIONS	nothing guards changes or advances work items unattended	automations.md, automations-security.md

ACTIONS — the callable surface against a running service. One action is the smallest meaningful business outcome composed of several calls, not an endpoint wrapper; the worked create-order example is the load-bearing rule. Tribal knowledge (auth, secret loading, headers, FK resolution, test-data markers) is encoded once so no session rediscovers it. Verbose by default, because the observed trace is the fixture the later automated test is written from. Secrets redacted on every output path.

ENVIRONMENT — the isolated ephemeral environment. Empty schema ≠ runnable; data is part of the environment. Every dependency leaving the box is triaged safe or unsafe, with uncertain counting as unsafe until the user decides otherwise, and six containment options ordered cheapest→most faithful that the user picks from. Proof is by attempting the call, never by reading config.

PROMPTING — the skills and subagents an AI coding agent loads (Claude Code, Codex, Cursor, Copilot, Windsurf, Antigravity, Devin). Organized around the contract a skill makes — when I run, what I read, what I do, what I refuse, what "done" means — with 14 blocks across those five clauses; a gap picks four or five. Two rules carry most of the weight: anything holding procedure, capability, knowledge, or scripts is a skill, while a subagent is a context placeholder holding a non-specialized proto-role, so there is no skill-versus-subagent decision to make. And procedure when order is the correctness, guidance when judgment is — a hardcoded branch where judgment belongs produces confident wrong answers. Proof triggers are derived from invariants rather than improvised, and routing is asserted first, because a skill that never loads is the most common failure and the happy path hides it.

HOOKS — lifecycle hooks for AI coding agents. Prompts persuade, hooks enforce: a rule is advice the model may skip, a hook is code the runtime runs at a fixed event regardless of what the model decided. Five capability classes — intercept and block, rewrite tool input before execution, rewrite tool output before the model sees it, inject context, observe.

The asset routes to seven empirically verified per-agent contracts, because the wire contracts diverge in ways that fail silently: two independent block mechanisms that must never be mixed, post-execution events that cannot block because the tool already ran, user-facing and model-facing text in different fields so the wrong one reaches nobody, structured output parsed only on one exit code, and one agent with no structured-output channel at all that blocks via stderr and exit 2. Ships each agent's working hooks.json, the live-run logs as grep targets with the marker set named, and scripts/tester.js — a universal dump-first probe that discovers an undocumented contract by wiring it and reading what arrives.

AUTOMATIONS — work that happens without a human driving it, for prevention (guards: decides allow/flag/block, never authors what it judges) or autonomous execution (triage, docs, planning, small implementations, reconciliation). Different blast radius — a false block versus a wrong change landing — so different trust postures and permission sets.

The substrate is a resolved decision, not the kind: CI pipeline, scheduled job, workflow automation, process engine, bot or webhook service, hook or policy controller — chosen on where state lives, how long the work runs, whose identity it uses, what it must reach, and who approves.

The state model is the design. Three states per automation with the terminal state never equal to the input state, so re-processing is structurally impossible rather than merely discouraged; the claim into a working state is the concurrency lock and the only visible running signal; a crashed run parks in a state nothing loads from and waits visibly instead of looping; and every flow has at least one human task, so no automation shows only its end result. A process engine formalizes exactly these as user tasks — the invariants were never CI-specific.

Automation security

Everything the triggering system hands over is untrusted data — on a forge: titles, bodies, comments, branch names, file names, file contents. The definition and the router prompt load from the trusted baseline; a proposed change is inspected, never executed in a job holding credentials. The trigger that grants a write token and secrets while checking out unmerged code is the standard vulnerability of this category, and the asset says to name yours and prove it is not that.

Write permission is scoped per state, not per automation. Secrets do not belong in the model's job at all — privileged steps go in a separate job that runs no model. The agent's own output is an attack surface, since injected text in its comment reaches a human or a downstream automation. Published transcripts are scrubbed; they sit outside log masking. And guardrails must live outside the agent's write reach — branch protection, or the substrate's equivalent, is the mitigation; the prompt asking nicely is not.

"Internal" is not private. An internal repository is public to the whole organization — employees, contractors, associates. Whether the actor is internal or external changes nothing; only the result does, measured as damage to the project, the team, the organization. A detection therefore routes to a channel narrower than the item that carried it: a world- or org-readable alert republishes the attack and tells the actor it worked.

automations-security.md ships a guardrail starter copied verbatim into the target repo's router prompt: threat signals, the rule that framing ("test", "red team", "demo") grants no exemption, the stop-and-alert protocol, and a non-public-findings section covering what may and may not appear in a world-readable comment.

Files

File	Purpose
skills/harness/SKILL.md	Router, classification, HITL gate, handoffs, inventory
skills/harness/assets/*.md	Eight assets, per the table above
skills/harness/references/hooks/*	Seven per-agent hook contracts, working configs, live-run logs (copied)
skills/harness/scripts/tester.js	Universal hook probe
skills/harness/references/configure/*.md	Per-coding-agent formats and feature support (copied)
skills/harness/README.md	Maintainer doc
docs/definitions/skills.md	Registers harness
skills/testing/SKILL.md, skills/coding/SKILL.md	Loose "CLI testing harness" lines now point at harness
plugins/**	Regenerated, both profiles, 14 variants

Decisions

Question	Decision
Invocation contract	Model-invocable skill; no disable-model-invocation. The frontmatter description carries every activation trigger
Secrets	Redacted by default on every output path; --show-secrets opt-in and gated by sensitive-data
Environment boundary	Local/isolated by default; shared or higher environment is an HITL gate via risk-assessment / dangerous-actions
Inventory format	## Harness section in the target repo's ARCHITECTURE.md, written by SKILL.md step 9 alone
Harness placement	Discovered per project and proposed via HITL; no hardcoded path
PROMPTING scope	Runtime execution and assertion. Text-level lint — frontmatter, section names, dead references — is separate and no substitute for a run
Implementation handoff	coding-flow for ACTIONS, ENVIRONMENT, HOOKS, AUTOMATIONS. PROMPTING authors inline; that flow's reviewer and validator phases duplicate the harden and prove steps its assets already own. An automation's router prompt is authored through the PROMPTING assets
AUTOMATIONS genericity	Teaches decisions and invariants. No state names, no tracker, no substrate assumed — the substrate is resolved per project
Guardrail starter	Shipped verbatim as a runtime artifact to copy; only the alert channel and forge command are meant to be adapted
Hook references	Copied from docs/hooks/: prose paths repointed, unshipped .jsonl pointers dropped, adapter source path replaced with a raw GitHub URL, Rosetta-internal phrasing neutralized. JSON payload captures left verbatim — they are run evidence
Target of PROMPTING and HOOKS	Named explicitly as AI coding agents. Without naming them the skill cannot tell what class of artifact is meant
configure/	Copied verbatim into references/configure/, reached with APPLY SKILL FILE. Root folder and the READ CONFIGURE alias still ship — retiring both is follow-up work

Design notes

• No <when_to_use_skill>. Only the frontmatter description is visible before a skill loads, so a post-load trigger list is dead weight. Same omission as orchestration.
• Registration is single-source. No asset carries a <registration> block; SKILL.md step 9 owns the ## Harness section name.
• Step 7's per-kind clauses are load-bearing. Without the PROMPTING exemption and the AUTOMATIONS split, the router contradicts itself for two of its four kinds.
• Boundary note for reviewers: the assets name guardrail skills (sensitive-data, risk-assessment, dangerous-actions) directly — a deliberate deviation from "skills don't call skills," matching the existing prerequisite pattern in orchestration and rosetta. No asset deep-links another skill's internals.
• Self-conformance. The assets are held to the rules the prompting ones teach — layered lines, no meta-explanation, nothing a competent model already knows.
• Precedent: src/rosetta-mcp-server/validation/verify_mcp.py for the service kinds; tests/e2e-tests/* and docs/manual-tests/*.md for PROMPTING; this repo's own docs/AUTOMATION-ARCHITECTURE.md, .github/workflows/repo-*.yml, .github/prompts/ and .github/scripts/ for AUTOMATIONS — where the state-model invariants were measured, and one project's answers rather than the shape the asset prescribes.

🤖 Generated with Claude Code