Package the portable Skill as a plugin

[raghubetina]

Summary

• expose the canonical portable Skill through a narrow source-only marketplace plugin without duplicating instructions or runtime dependencies
• keep a distinctly named repository-root one-session preview separate from the SHA-versioned marketplace source
• add structural CI guards and a manual isolated-install release smoke with exact component, cache, and state boundaries
• make strict CLI validation and live-versus-reviewed observation drift executable release gates
• document unreleased CLI/service gates and dated machine-readable packaging evidence

Verification

• Node 22.0.0 and 24.18.0: 36/36 each
• marketplace manifest and root preview manifest both pass strict validation
• isolated install: exact 8 files / 241,779 bytes, 1 combined Skill/Commands, zero Agents/Hooks/MCP/LSP, derived Commands absent, no PATH package manager, scoped monitored state unchanged
• dangling links cannot masquerade as absent isolated paths
• exact CLI contract 7944bf3c
• npm audit 0
• source and evidence checks clean

Boundaries

Source packaging only. No real profile install, plugin/Skill/CLI publication, GitHub clone, model-backed session, or live service journey.

[raghubetina]

Review

The packaging split holds up and the verification claims reproduce. Two notes below, one a design question about the version pin and one about the precision of the audit claim. Neither blocks.

Verification

Check	Result
npm run check (Node 24.18.0)	36 tests, 36 pass
claude plugin validate --strict .claude-plugin/plugin.json	passed, reported as plugin manifest
claude plugin validate --strict .	passed, reported as marketplace manifest
npm run check:claude-plugin-install	every behavioral field matched; failed only on the pinned CLI version
npm audit --omit=dev	0 vulnerabilities

My first test run failed with Cannot find package 'ajv' because I had not installed yet. After npm ci the count matches the description exactly.

On the install smoke, the live observation agreed with the committed evidence on all of it: 8 files, the per-file byte sizes and SHA-256 digests, the installed tree digest, skillsAndCommands: 1, and agents, hooks, lspServers, mcpServers all zero. The single diff was version: '2.1.221' locally against '2.1.220' recorded.

Mutation check

I replaced lstatSync with statSync in pathEntryExists (script/plugin-isolation.mjs:144), importing statSync so the mutation tested semantics rather than producing a syntax error. The dangling-link test failed, so that guard has real coverage rather than being decorative.

The distinction it protects is worth stating: statSync and existsSync follow symlinks, so a dangling link at a monitored path reads as absent. lstatSync stats the link itself. Since the gate asserts that certain isolated paths stay absent, the follow-the-link version would let a leftover dangling link pass as clean state. Rethrowing anything that is not ENOENT rather than treating it as absence is the right companion choice.

The version pin is redundant with the fields around it

reviewedPackagingObservation (script/claude-plugin-observation.mjs:148) passes claudeCode through whole, so version sits inside the assert.deepEqual. Any Claude Code release fails the gate and sends the operator through re-recording, renaming the dated evidence, and updating the reviewed pins.

That fired for me one day after the evidence was recorded, on a patch bump, with zero behavioral difference.

The evidence note gives the rationale at evidence/2026-08-02-claude-code-plugin-install-smoke.md:83: Agent discovery reads Markdown under agents/ or explicit manifest paths, that rule is version-sensitive, so a recording should be rerun on upgrade. The reasoning is sound. The gap is that componentInventory is already compared strictly, so the risk it names is already caught directly. If a future version started discovering agents/openai.yaml, agents: 0 would become agents: 1 and the gate would fail on that field whatever the version said.

So the version equality adds churn without adding protection against the stated risk. What it could still catch is a version change that alters behavior the observation does not record, but the observation covers the file inventory, the digests, the tree digest, and the component counts, so that residue is small.

Worth considering recording version as context while asserting on the behavioral fields. Frequent red gates on a check whose remedy is "re-record" tend to train the reflex that would mask the genuine drift this is built to catch. Your call on whether the strictness is worth the churn; flagging because the trade looks different once you see it fail on a patch bump.

The audit claim needs a qualifier

"npm audit 0" holds for npm audit --omit=dev. Plain npm audit reports one high severity advisory, fast-uri reachable only through ajv:

@firstdraft/skills@0.0.0
└─┬ ajv@8.20.0
  └── fast-uri@3.1.4

ajv is already in devDependencies on main, so this PR does not introduce it, and the diff to package.json only adds the two scripts. The package ships no runtime dependencies, so nothing reaches an installed plugin. Worth stating the qualifier in the description so a later reader does not take the bare claim and stop looking.

What holds up

The two-manifest split does what the description says. firstdraft-preview at the repository root and firstdraft inside the marketplace are distinctly named, so the one-session preview and the installed firstdraft@firstdraft-skills cannot collide in a profile that has both.

Deriving Commands absence rather than reading it is the right call, and the evidence note is explicit that the CLI folds Skills and Commands into one group, so skillsAndCommands: 1 is not a Commands count. Deriving it from the manifest having no Commands declaration plus the exact installed file set is a claim the observation can actually support.

Real-state presence staying run-local rather than compared across machines is the right boundary, and requiring a core registry target on every run keeps that from silently degrading into no check at all.

[raghubetina]

Testing against a tool you do not control

Most tests you write check your own code. You call a method, you assert on what comes back, and if it changes you changed it. This PR is doing something different, and the difference is what makes it interesting.

The question it has to answer is: when someone installs this plugin through Claude Code, what actually lands on their disk? That answer is not produced by any code in this repository. It is produced by the Claude Code CLI, which is somebody else's program, on a release cadence nobody here controls.

You cannot unit test that. What you can do is three things, and this PR does all three.

One: write down what you saw

The pattern is a characterization test, sometimes called a golden file or an approval test. You do not assert that the output equals something you reasoned out in advance. You run the real thing once, record exactly what happened, commit that recording, and from then on assert that it still happens.

Here the recording is evidence/claude-code-plugin-install-observation.json, and it holds the per-file byte sizes, a SHA-256 for each file, a digest of the whole installed tree, the CLI's component counts, and the validation results.

The distinction between the two npm scripts is the whole idea:

"check:claude-plugin-install": "node script/check-claude-plugin-install.mjs",
"record:claude-plugin-install": "node script/check-claude-plugin-install.mjs --observation-output evidence/..."

Same script. One asserts against the recording, the other replaces it. Re-recording is a deliberate act with a visible diff, not something a test run does quietly on your behalf.

The reason this earns its keep: nobody had to predict that installing this plugin yields exactly eight files. They installed it, looked, and wrote down the eight. Now a future Claude Code release that starts pulling in a ninth file shows up as a failing check rather than as a surprise in somebody's profile.

Two: keep the test out of your real config

A test that installs a plugin has an obvious hazard. Installing normally writes to your actual Claude Code configuration and plugin cache. A test that does that has stopped being a test and started being a change to your machine.

So the run installs into an isolated cache, and then it verifies that your real state did not move:

function assertRealStateUnchanged(stage) {
  const changedRealState = changedRealStateEntries(realStateBefore);
  if (changedRealState.length === 0) return;
  ...
  throw new Error(
    `real Claude configuration or plugin cache changed ${stage}: ` + ...
      "Inspect the named paths, then remove any escaped user-scoped state with:\n" +
      "  claude plugin uninstall firstdraft@firstdraft-skills --scope user\n" + ...
  );
}

Two details worth stealing. It snapshots before and compares after, rather than trusting that isolation worked. And when isolation fails it hands you the exact cleanup commands, because the person reading that error is in the middle of a mess and does not want to go read documentation.

There is a sharp edge hiding in "did this path stay absent," and the PR handles it:

export function pathEntryExists(target) {
  try {
    lstatSync(target);
    return true;
  } catch (error) {
    if (error.code === "ENOENT") return false;
    throw error;
  }
}

lstatSync, not statSync and not existsSync. The two ordinary choices follow symlinks, which means a symlink pointing at nothing reports as absent. That is exactly wrong for this question. Something was created at that path, and the check exists to notice things being created at that path. lstat looks at the link rather than through it.

I confirmed this is load bearing by swapping in statSync and rerunning: the test named for it fails. Also notice the catch rethrows anything that is not ENOENT. A permissions error is not evidence of absence, and pretending it is would turn a broken check into a passing one.

Three: derive what the tool will not tell you

This is the subtlest part, and it is the one most worth internalizing.

The plugin needs to prove it ships no Commands. The natural move is to ask the CLI for a Commands count. You cannot, because the CLI reports Skills and Commands together in one group. What comes back is skillsAndCommands: 1.

That number is compatible with one Skill and no Commands, which is the truth here. It is equally compatible with zero Skills and one Command. Reading it as a Commands count would be reading a fact that is not in the data.

So the evidence note spells out what the derivation actually rests on: the manifest declares no Commands, and the installed file set is exactly the eight known files. Together those support the conclusion. The live count does not, on its own, and the note says so rather than letting the number imply more than it carries.

That habit generalizes well past this repository. When a tool gives you an aggregate and you want a component of it, the aggregate is not the answer. Either find independent evidence for the piece you want, or say plainly that you do not have it.

The part where this gets awkward

Recording what a third-party tool does means your test now has an opinion about that tool's version. This PR puts the version inside the comparison, so a Claude Code release makes the check fail.

There is a real reason for that. Whether agents/openai.yaml counts as an Agent depends on Claude Code's discovery rules, and those rules can change between versions. Forcing a human to re-verify after an upgrade is a defensible instinct.

The cost is that it fires constantly. I hit it on a patch bump one day after the evidence was recorded, with every behavioral field identical.

Worth sitting with, because the tension is general. Pin loosely and you miss the drift you built the check for. Pin tightly and the check goes red so often that re-recording becomes muscle memory, which is the same as not having the check. The way out is usually to ask which field would actually move if the thing you fear happened, and assert hard on that one. Here, a version that started discovering that YAML file would change the Agents count from 0 to 1, and the Agents count is already compared strictly.