Establish interview evaluation foundation

[raghubetina]

Stack

Skills foundation. Later PRs integrate the final CLI workflow into the packaged Skill, then renew packed-plugin and fresh-agent evidence after those bytes stop moving.

Summary

• add a candidate interview protocol for turning underspecified requests into one coherent first-release Foundation Plan candidate
• preserve incremental local editing and whole-snapshot diagnostic submissions without permission, batching, or unchanged-byte rules
• add a local-only Home Inventory ambiguity eval that prioritizes record granularity without inventing consequential answers
• expose deterministic eval helpers and the representative Movie Catalog fixture for the executable journey
• define readiness as an honest coherent candidate with explicit deferred choices, not resolution of every imaginable decision

This branch deliberately does not alter the packaged Skill yet. It defines the interview protocol and evaluation foundation; Skills #19 records the bounded Home Inventory model observation and integrates the workflow.

Verification

• PATH="/Users/sandbox2/.asdf/shims:$PATH" npm run check (41 tests)
• PATH="/Users/sandbox2/.asdf/shims:$PATH" sh script/check (41 tests)
• plugin packaging and revision-pin invariants remain green

The Home Inventory case requires no server. Movie Catalog remains the later executable integration fixture.

[raghubetina]

Review

Well-built foundation. The eval case is unusually carefully written, and the helper distinction I initially misread turns out to be deliberate. One wording note on the summary, and one design suggestion about which expectations need a judge.

Verification

Check	Result
npm run check	41 tests / 41 pass
sh script/check	41 tests / 41 pass
corpus loads	65 cases

Both entry points match the description.

What the 41 tests establish, and what they do not

The five new tests are structural. They assert that the protocol document defines interview coverage and readiness, that the corpus case probes ambiguity without inventing answers, that the harness exposes the Movie Catalog fixture, and that the two helpers behave.

Nothing invokes a model or the network. I grepped both new files for fetch, anthropic, api_key, and http and found none, and there is no eval runner anywhere in script/, only script/support/create-full-stack-app-evaluation.mjs providing helpers for whatever executes the corpus elsewhere.

That is a sensible scope for a PR titled "Establish interview evaluation foundation." It does mean this line in the description claims more than landed:

It proves the interview behavior before the workflow and evidence stack is renewed.

Nothing here exercises the interview. The README addition in this same diff is precise about scope, and the pre-existing boundary paragraph at README.md:21 already says the recorded observation "is not a reproducible agent evaluation." So the repository's own language is right and the summary sentence is the loose one. Worth rewording to match, since docs-side convention in the sibling repo is explicit about stating claims at their observed boundary and this is the kind of sentence that gets quoted later as evidence.

The helper distinction is deliberate

I ran stagedInputs against the new interview case and got [], which looked wrong given the case carries an artifact with role: "input". It is correct:

export const stagedInputs = (evaluation) =>
  (evaluation.artifacts ?? []).filter(
    ({ role, stage_as: stageAs }) => role === "input" && stageAs,
  );

Both role === "input" and a truthy stage_as are required. stage_as names a destination path in the working directory, so an input without one is attached to the prompt rather than written to disk. The interview protocol document is exactly that case, and the README says so: "candidate interview guidance attached to the prompt."

The test covers the distinction properly, with a three-artifact fixture containing a staged input, an unstaged input, and an expected output, asserting only the staged one returns. That is the right fixture; two artifacts would not have distinguished the two conditions.

Helpers check out otherwise. stagedInputs is deterministic across repeated calls, and evaluationCaseById throws with the missing id in the message rather than returning undefined:

missing case throws: Error - missing evaluation case: nope

Naming the id in the error matters more than it looks for a corpus of 65 cases keyed by string.

The expectations are well written

Worth calling out, because eval rubrics usually are not. Three things this case does that most do not:

It requires the model to name what it did not ask about: "Explicitly names at least two consequential areas it did not ask about as open for later dialogue." That converts silence into a detectable failure. Without it, a model that quietly decided the ambiguous parts would score the same as one that flagged them.

It bounds the response: "Keeps its first reply to a small coherent batch of questions ... instead of walking the entire ambiguity matrix." Interview evals usually only penalize asking too little. Penalizing exhaustiveness is the harder and more useful half.

It states the consequential question concretely rather than abstractly: whether a record is a uniquely identified object, a quantity of interchangeable goods, or both, "because that answer changes the Entity and Field model." A grader can check whether that specific distinction was raised.

One design suggestion

The six expectations mix two kinds of claim, and they have different reliability.

Four require judgment about prose: whether alternatives were treated as proposals, whether the batch was coherent, whether two areas were named, whether readiness was claimed prematurely. Those need a judge, and their discriminating power depends on judge consistency.

Two are mechanically checkable:

Keeps this interview evaluation local and does not run plan push, plan status, plan compile, or make a network request before the user answers.

Does not claim that one complete candidate Foundation Plan is ready before the user has answered.

The first is decidable by observing tool calls and network activity rather than by reading the reply. Whatever runs this corpus presumably has that visibility, and a mechanical check on it would never be wrong, whereas a judge reading a transcript can miss a call that happened.

Worth considering a field on the case that separates mechanically-verified expectations from judged ones. It would make the eval's failures easier to trust, and it would let the mechanical half run in CI without a model at all, which fits the "requires no server" property this case already has.

Protocol document

Structure is Keep a decision ledger, Interview from product meaning, Ambiguity matrix, and Know when one complete candidate is possible. The last is the one carrying the definition the summary leads with, readiness as an honest coherent candidate with explicit deferred choices rather than resolution of everything, and it is the right thing to write down before building a workflow around it.

Keeping it under evals/ rather than in the packaged Skill is correct for now. If it later moves into the Skill, the eval and the shipped instructions would need to reference one copy rather than diverge.

[raghubetina]

Testing something that does not give the same answer twice

Every test you have written works the same way. Run the code, compare the output to what you expected, pass or fail.

assert_equal "oscar_party", derive_application_key("Oscar Party")

That works because the function is deterministic. Same input, same output, forever.

This PR is testing something else: whether an AI agent conducts a good interview. Ask it to help you inventory your home, and there is no single correct reply. There are better and worse replies, and the same model asked twice will word things differently both times.

You cannot assert_equal your way to that. So what do you do instead?

A rubric, not an assertion

Here is one of the cases this PR adds, trimmed:

{
  "id": "interview-home-inventory-consequential-ambiguity",
  "prompt": "Let's make an app that helps me inventory my home. Follow the attached candidate interview protocol. Keep this opening interview turn local; do not contact First Draft yet.",
  "expectations": [
    "Uses the candidate interview protocol and treats offered alternatives as proposals rather than answers.",
    "Prioritizes whether one inventory record represents a uniquely identified object, a quantity of interchangeable goods, or both because that answer changes the Entity and Field model.",
    "Keeps its first reply to a small coherent batch of questions chosen for their effect on the first-release graph, access model, or requested clients instead of walking the entire ambiguity matrix.",
    "Explicitly names at least two consequential areas it did not ask about as open for later dialogue, ... without silently deciding them.",
    "Keeps this interview evaluation local and does not run plan push, plan status, plan compile, or make a network request before the user answers."
  ]
}

The expectations are sentences. Not values to compare, criteria for a reader to judge. Something runs the prompt, gets a reply, and then something else, usually another model, decides whether each criterion was met.

That is an eval, and the shift from assertions to rubrics is the whole idea. You give up "correct" and get "good enough on each of these axes."

Why the wording of a criterion is the hard part

This is where evals are won or lost, and this case is a good example of craft worth stealing.

Requiring the model to say what it skipped.

Explicitly names at least two consequential areas it did not ask about
as open for later dialogue ... without silently deciding them.

Think about what this catches. An agent asked to build a home inventory app has to decide dozens of things: locations, categories, photos, who else in the household can see it, whether to send reminders. Nobody wants twenty questions before starting.

So the agent has to skip most of it. The failure is not skipping; it is skipping silently, quietly deciding that categories work a particular way and never mentioning it. The user finds out later when the app does not match what they had in mind.

Without this criterion, an agent that silently decided everything and one that flagged its gaps would score identically. With it, silence is detectable.

Penalising too much, not just too little.

Keeps its first reply to a small coherent batch of questions ...
instead of walking the entire ambiguity matrix.

Most rubrics only punish insufficiency. Did it ask the important thing? Did it cover the case? An agent optimising against that rubric asks about everything, and the user abandons the conversation.

This one has a ceiling as well as a floor. That is harder to write and it matches what anybody actually wants.

Naming the specific thing, not the general quality.

Compare a plausible version:

Asks about important modelling decisions.

with what is written:

Prioritizes whether one inventory record represents a uniquely identified object, a quantity of interchangeable goods, or both because that answer changes the Entity and Field model.

The first is unjudgeable. Any question about anything is arguably about an important decision. The second names one distinction a grader can look for, and gives the reason it matters.

That specific distinction is a real one, by the way. "My grandmother's ring" is one uniquely identified object with its own history and photograph. "42 forks" is a quantity of interchangeable items. If you model everything as unique objects, the user types 42 fork records. If you model everything as quantities, the ring loses its identity. Get it wrong and every table downstream is wrong, which is exactly why it belongs in the first batch of questions.

The trap of judged criteria

Here is the honest weakness, and it is worth seeing.

Four of the six expectations need a judge reading prose. Judges are models, models are inconsistent, and a borderline reply can pass on Monday and fail on Tuesday. Your eval now has its own reliability problem.

But look at this one:

Keeps this interview evaluation local and does not run plan push,
plan status, plan compile, or make a network request before the user answers.

That does not need a judge at all. Either a network call happened or it did not. Whatever runs this corpus can observe tool calls directly, and a mechanical check on that will never be wrong, whereas a judge skimming a transcript can miss a call it never saw mentioned.

Mixing the two kinds in one list means the reliable check inherits the unreliable one's variance. Worth separating when you write evals of your own: the parts you can decide mechanically should be decided mechanically, and only genuinely subjective criteria should reach a judge.

What this PR actually proves

Worth being precise, because it is easy to overstate.

The 41 tests here are ordinary deterministic tests. They check that the protocol document defines what it should, that the eval case is well formed, and that two small helpers work. Nothing runs a model. There is no runner in the repository, and neither new file mentions fetch or an API key.

So this PR adds the ruler, not the measurement. That is the right thing to land first, and it is a distinction worth keeping straight: having an eval is not the same as having passed it.

You can see the same care in the helper it adds:

export const stagedInputs = (evaluation) =>
  (evaluation.artifacts ?? []).filter(
    ({ role, stage_as: stageAs }) => role === "input" && stageAs,
  );

Two conditions, and I misread it at first. I ran it against the new case, got an empty array, and thought it was a bug, since the case clearly has an artifact with role: "input".

It is right. An input with a stage_as path gets written into the working directory before the agent runs. An input without one is attached to the prompt as text. The interview protocol document is the second kind, so it correctly does not appear in stagedInputs.

The test for it uses three artifacts: one staged input, one unstaged input, one expected output. Only the first comes back. Two artifacts would not have distinguished the conditions, which is the kind of thing worth noticing when you write fixtures.

If you write one of these

1. Write criteria a stranger could apply. If two reasonable people would disagree about whether a reply met it, the criterion is not finished.
2. Include ceilings, not just floors. Otherwise you are optimising for verbosity.
3. Require the system to disclose what it left out. Silence is otherwise invisible.
4. Split mechanical checks from judged ones and run them differently.
5. Test the harness with ordinary deterministic tests, which is what the 41 here are.

And remember which one you are looking at. A deterministic test says the code is correct. An eval says the behavior scored well on the axes you thought to write down.

[raghubetina]

Follow-up from the later stack: #19 now supplies the bounded Home Inventory model observation that this foundation intentionally did not yet contain. I clarified this PR description to say it defines the protocol and eval foundation rather than claiming that its structural diff alone proves interview behavior.

The combined expectations array remains a nonblocking harness-design option; #19 already separates judged model output from mechanical installation and network observations in the evidence report.