Record workflow evaluation evidence

[raghubetina]

Summary

• Align the authoring eval corpus with the relaxed workflow: incremental local drafting, diagnostic pushes whenever useful, independent repairs beside preserved capability gaps, and pragmatic rather than prohibitive Compile guidance.
• Preserve a local Home Inventory opening-interview response and one bounded Movie Catalog diagnostic-to-Compile rehearsal as privacy-safe, revision-pinned evidence.
• Refresh the isolated Claude Code 2.1.221 plugin installation observation, documented discovery review, exact installed inventory, and deterministic evidence assertions.

Integration

CLI #17, #16, #18, and #19 and Skills #17 and #18 are now on their default branches. The reviewed CLI evidence revision f55edffc9e88924f9a4c95f41c4d0bc9b72422f8 remains in CLI main ancestry, so the historical evidence and exact digest remain unchanged. This PR now targets Skills main.

The service rehearsal remains revision-pinned through firstdraft/firstdraft #306. This PR does not implement firstdraft/firstdraft #291, release the plugin or CLI, contact staging or production, or create a real GitHub repository.

Verification

• Current integrated head: sh script/check: 42 tests, 42 passed; exact external contract passed against CLI f55edffc9e88924f9a4c95f41c4d0bc9b72422f8; git diff --check origin/main...HEAD: passed.
• npm run record:claude-plugin-install: both exact strict validations passed; isolated installation contained 8 canonical Skill files and 207,433 bytes; live inventory was Skills 1, Agents 0, Hooks 0, MCP servers 0, and LSP servers 0; monitored real state was unchanged and temporary state was removed.
• Skill Creator quick validation: passed.
• Model rehearsal: native Claude Code 2.1.221, Opus/high, service 3a029a8b425addbbba4f56d9197878cc002752f4, candidate plugin b5c3897b240bfa3a9117d1a564d8e6b7d783e993, freshly packed CLI f55edffc9e88924f9a4c95f41c4d0bc9b72422f8; two pushes, two bounded status waits, one Compile; final valid graph version 2; 194 files and 542,894 bytes verified through strict fake GitHub publication, retained download, and materialization.
• Service #306 final hosted run 30946058871: all six jobs passed for the historical evidence revision.

Evidence limits

Home Inventory is an opening-turn local interview evaluation, not a compilable live fixture. Movie Catalog is one constrained local rehearsal with fake GitHub transport; it does not establish installed-marketplace model use, published distribution, arbitrary applications, generated-app execution, deployment, real GitHub, staging, or production readiness. The CLI and plugin remain unreleased.

[raghubetina]

Hosted Node 22 and Node 24 checks stop at the intentional external-contract ancestry gate: f55edffc9e88924f9a4c95f41c4d0bc9b72422f8 is not yet an ancestor of the CLI default branch. The exact contract check passes locally against the stacked CLI worktree. After CLI #17, #16, #18, and #19 land, update this stack from its base and rerun hosted CI; no Skills code failure is currently hidden behind this gate.

[raghubetina]

The prerequisite stack is now landed: the pinned CLI evidence revision remains in main ancestry, Skills #17 and #18 are on main, and the refreshed Node 22/24 run passes both jobs. The earlier ancestry-gate note is resolved; no repin or evidence rewrite was needed.

[raghubetina]

Review

Reviewing after the merge, since this landed at 22:09Z. Two things I raised on earlier branches resolve here, and one recurs.

Verification

Check	Result
sh script/check	42 tests / 42 pass
installed inventory	8 files, 207,433 bytes, exact match
component inventory	Skills 1, Agents 0, Hooks 0, MCP 0, LSP 0, exact match
both strict manifest validations	passed

The byte total reconciles exactly against the per-file sizes in the observation.

This closes the finding I raised on Skills #18

On #18 I flagged that the packaged Skill still instructed three commands the CLI stack had removed, at SKILL.md:126, :230, and :348. That was accurate for #18, which did not touch skills/. This branch is the one that fixes it, and it does so better than by deleting the references.

The mentions that remain are now negative:

There is no public `plan publish`, `plan subject-id`, or `plan compile --output` contract. Do not install,
download, or upgrade the unreleased CLI automatically. If a required command is missing, report that capability
gap rather than approximating it with direct HTTP.

I confirmed no instruction to run any of them survives anywhere under skills/, and that the listed surface is now exactly the current one: generate uuid, generate application-key, plan init, plan push, plan status, plan compile, compilation status, compilation download.

Stating what does not exist beats silence. An agent carrying stale expectations from anywhere else gets told directly rather than discovering it through Unknown command.

The last sentence is the best line in the diff. "Report that capability gap rather than approximating it with direct HTTP" gives a missing command a defined outcome. Without it, the plausible agent behavior on a missing command is to reach for curl, which would bypass every validation the CLI performs.

The version pin fired again, one day later

On #16 I noted that putting claudeCode.version inside the strict comparison would fail on any patch release, and that it had already done so once. It has now happened a second time in two days:

Branch	Recorded	Mine at review time
#16	2.1.220	2.1.221
#19	2.1.221	2.1.222

Running npm run check:claude-plugin-install here:

AssertionError: live Claude Code plugin observation differs from committed evidence
+     version: '2.1.222'
-     version: '2.1.221'
      fileCount: 8,

Every behavioral field matched: 8 files, agents: 0, both validations passing. The version was again the only difference.

Two forced re-records in two days, neither reflecting any change in what the check exists to detect, is a different proposition from the one-off I described on #16. The remedy the assertion message prescribes is re-record, rename the dated evidence, update the pins, which is real work each time and trains exactly the reflex that would wave through a genuine drift.

The argument from #16 still stands: componentInventory is compared strictly, so a Claude Code release that started discovering agents/openai.yaml would move agents from 0 to 1 and fail the gate on that field regardless of the version. Recording the version as context rather than asserting on it would keep the protection and stop the churn.

Worth deciding now rather than after the fifth re-record.

The Skill got substantially shorter

Comparing the installed inventory against #16's:

File	#16	#19
SKILL.md	33,276	15,272
references/diagnostics-and-recovery.md	34,185	13,600
total	241,779	207,433

SKILL.md at less than half its previous size is a large change to ship alongside everything else here. Shorter instructions are generally better followed, and the removed material is presumably the obsolete publish and local-compile contract plus the prohibitive Compile guidance the summary mentions relaxing.

The check on whether anything load bearing went with it is the eval corpus and the model rehearsal, and both are exercised. Worth noting for the record that this is the largest single reduction in the shipped instructions so far, so if a future rehearsal starts failing in an unexplained way, this is the change to look at first.

Evidence limits

The final section is exemplary and worth keeping in that form:

Home Inventory is an opening-turn local interview evaluation, not a compilable live fixture. Movie Catalog is one constrained local rehearsal with fake GitHub transport; it does not establish installed-marketplace model use, published distribution, arbitrary applications, generated-app execution, deployment, real GitHub, staging, or production readiness.

Nine explicit non-claims attached to evidence that could easily be read as more than it is. That is what makes the evidence usable later rather than something a reader has to re-derive the scope of.

The integration note is similarly precise: the pinned CLI revision f55edffc remains in main ancestry, so the historical evidence and digest stay valid rather than needing re-recording alongside everything else.

[raghubetina]

Writing down what does not exist

There is a line in this PR that looks like a leftover and is doing real work:

There is no public `plan publish`, `plan subject-id`, or `plan compile --output` contract.

Three commands that were removed from the CLI. You might expect the instructions to simply stop mentioning them. Instead they mention them by name, to say they are gone.

That choice is worth understanding, because it applies to any documentation an AI agent reads, and increasingly that is a lot of documentation.

The problem with deleting

The obvious cleanup after removing a feature is to delete every reference. The docs then describe what exists, and nothing more. Tidy.

For a human reader, fine. A human who tries a removed command gets an error, shrugs, and looks at --help.

For an agent, silence is ambiguous in a way it is not for a person. The agent may have picked up plan publish from somewhere else entirely: a blog post in its training data, an older version of these instructions, a pattern it inferred from similar CLIs. Your document not mentioning it does not tell the agent it does not exist. It just fails to mention it.

So the agent tries. Gets Unknown command. And now has to decide what that means, with no guidance, which is where things go wrong.

The three things an agent does next

Watch what happens when an agent hits Unknown command with no instruction covering it:

Retry with variations. Maybe it was plan:publish, or publish plan, or needs a flag. You have now burned several turns and possibly done something unintended along the way.

Try to fix the environment. The command is missing, so the tool must be out of date. Install a newer version, or a different one. This is why the same paragraph adds "Do not install, download, or upgrade the unreleased CLI automatically." An agent that helpfully upgrades an unreleased tool has just changed the thing you were testing.

Route around it. This is the dangerous one. The agent knows what publishing means and can see an API. So it reaches for curl.

That last path is why the paragraph ends the way it does:

If a required command is missing, report that capability gap rather than approximating it with direct HTTP.

Read what this does. It gives "the command I need is missing" a defined outcome. Not an error to recover from creatively, an outcome: say so and stop.

That matters because the CLI is not a thin wrapper. It validates responses against exact key sets, verifies artifact digests against downloaded bytes, checks ETags for conditional writes. An agent that hand-rolls the same request with curl gets none of it, and the failure will not look like a skipped validation. It will look like everything worked.

The shape of the instruction

Three parts, and each closes off one of the three behaviors:

1. Name what is absent. Closes off retrying variations, because the agent knows it is not a spelling problem.
2. Forbid the environment fix. Closes off upgrading, which would invalidate the thing being tested.
3. Define the fallback. Closes off improvisation, by making "report the gap" the expected move rather than a failure to be avoided.

Without part three the first two just leave the agent stuck, and a stuck agent improvises. You have to give it somewhere to go.

The general principle

Instructions for an agent need a negative space: the things it might reasonably try that it should not.

Ordinary human documentation almost never has this. You do not write "this library has no flush method" in a README, because a human checks the method list. An agent working from instructions plus prior expectations benefits from being told which of its expectations are wrong.

Concretely, worth stating explicitly:

• capabilities that used to exist and no longer do
• capabilities that sound like they should exist and never did
• the fallback when something needed is unavailable
• actions that would technically work but bypass a safeguard

That last category is the one most worth writing down, because it is exactly where a capable agent's initiative causes damage. "You could do this and you must not" is not a sentence you write for humans. It is a necessary one here.

Shorter, not longer

One more thing worth noticing, because it cuts against how these documents usually evolve.

The instructions in this release got substantially smaller. SKILL.md went from 33,276 bytes to 15,272, less than half. The whole installed package dropped from 241,779 to 207,433 bytes.

That is the opposite of the normal trajectory. Instructions accumulate: every incident adds a warning, every misunderstanding adds a clarification, and nobody removes anything because removal feels risky.

The problem with accumulation is that instructions compete for attention. Guidance that is 40% obsolete is worse than half the length and current, because the reader, human or otherwise, has to work out which parts still apply. And when a command described in detail across several sections no longer exists, that detail is not neutral. It is actively misleading.

So the shrink here is doing two things: removing what is wrong, and making what remains more likely to be followed.

The negative statements survive the cut, and that is the interesting editorial judgment. Everything explaining how to use the removed commands is gone. One sentence saying they do not exist stayed. That is the right ratio: the detail was the liability, the correction is cheap and prevents a specific wrong turn.

If you maintain instructions an agent reads

• When you remove a capability, say it is removed rather than going quiet.
• Give "the thing I need is missing" a defined response, or improvisation fills the gap.
• Name the workarounds you do not want, especially the ones that would appear to succeed.
• Cut aggressively. Length is not thoroughness; it is competition for attention.
• Keep one line of correction where you deleted a page of description.