[0xMMA]

Depends on #127 (fix(llm): isolate Claude Code CLI calls from the user's sessions). This branch is rebased on top of it, so its first commit edad910 belongs to #127. Merge #127 first; after that this PR shows only its own commits.

What it does

Both eval suites (Pyramidize and Fix) run through the installed Claude Code CLI by default, and every sample is scored by two judges, claude-haiku-5-5 and claude-sonnet-5-5, each reported on its own. This is E3 phase 1, step "Instrument" (decisions of 2026-10-11).

CLI eval (the original #105, rebased):

• Pipeline and judges run as claude-code, so no API key is read.
• --provider claude and EVAL_JUDGE_PROVIDER=claude switch to the API. .env keys load only when an API provider is in the run.
• Resolved IDs: recorded with main's ModelRecorder (model holds the ID, plus modelRequested). Each judge records its own resolvedModel.
• CLI version: claudeCodeVersion is read at the start and end of a run and reported next to a verdict. It is not keyed (decision 2026-10-11), because the CLI updates itself.

Two judges:

• Records: summary.json has judges[], avgJudges and judgeCounts.
• Aggregate: eval-aggregate.sh and baseline.json hold one interval per judge, at run level and per sample, plus per-judge coverage. Nothing averages across judges.
  • Judge columns are keyed by the ID that answered, as the key is. Two spellings of the same judge (haiku and claude-haiku-5-5) land in one column.
• configKey: the judge provider, resolved model and temperature fields hold one +-joined entry per judge, ordered by ID.
  • A single-judge key is unchanged, so every recorded baseline reads "not comparable" (exit 2) against a two-judge run. A test runs every baseline in test-data/eval-baselines/ through this.
• --compare: applies the interval rule to each metric (deterministic, then each judge).
  • Any metric clearly worse is a regression. An improvement needs at least one metric clearly better and none worse.
  • regressedMetrics / improvedMetrics name what flagged the verdict, so a regression carried by one judge alone reads as that. The fields are filled only for a comparison that was actually made.
  • A judge with no interval on one side gives "not comparable", never "overlap".
• Overrides: EVAL_JUDGES=a,b,c chooses the set. Bridge run: EVAL_JUDGES=claude-haiku-5-5,claude-sonnet-5-5,claude-sonnet-4-5-20250929. EVAL_JUDGE_MODEL is a set of one. Both variables at once, an empty entry or a duplicate are refused.
• Temperature (llm.SupportsTemperature(provider, model)):
  • CLI: no judge is pinned (null plus a note).
  • API: Claude models from the 4.7 generation on take no temperature, so the 5.5 judges record null with a note instead of sending 0 and getting a 400. The Sonnet 4.5 bridge judge stays at 0.
• Judge IDs: Haiku 5.5 and Sonnet 5.5 have no dated snapshot, so the full ID is the pin. In the smoke run (CLI 2.1.296), modelUsage reported exactly those IDs.
• eval.sh:
  • Shows the judge set.
  • go test -timeout is 10800 s.
  • --dry-run asks the CLI nothing; --preflight adds the sign-in check.
  • Its env -u list is tested against blockedCLIEnv.

Review follow-ups (Opus review)

• Columns keyed by the requested name produced a silent wrong verdict (0.95 → 0.40 read "inconclusive", exit 0). Fixed and covered by a test.
• The API judge route sent temperature 0 to models that reject it. Now model-aware.
• API-era baselines: the docs no longer promise a route back to them. They are not comparable with this instrument on any route.
• 2026-10-04 CLI baselines: marked in both quality-status files as possibly affected by the auto-memory leak (fixed in fix(llm): isolate Claude Code CLI calls from the user's sessions #127) and superseded. The readings built on them ("the CLI route corrects less", the judge-spread explanation) are flagged as unconfirmed. Nothing was re-recorded.

Conflicts resolved against main

• Resolved model IDs: main's Response.Model, answeringModel and llm.ModelRecorder (feat: model family defaults, effort per feature, AI Providers redesign #108) are kept; the PR's ResolvedModels is dropped.
• Thinking: main's Request.NoThinking replaces the PR's DisableThinking. The CLI keeps ignoring it, as on main, so EVAL_THINKING / --thinking are gone. A Fix run records fixThinkingLabel: CLI on, fast-tier API off, otherwise n/a.
• Judge call: main's noEffort plus the PR's nil temperature and 180 s deadline.
• Unchanged from main: effort, model families, typed errors, max_tokens.
• CLI isolation moved to fix(llm): isolate Claude Code CLI calls from the user's sessions #127.
• Commit history: the original 11 commits are squashed into d471baf. That commit's message still describes the isolation, which now lives in fix(llm): isolate Claude Code CLI calls from the user's sessions #127; rewording it was blocked here. It does not matter after a squash merge.

Verification

• go vet -tags gtk3 ./... and go vet -tags eval,gtk3 ./internal/...: clean.
• go test -tags gtk3 ./internal/...: pass. Every new behaviour test was seen failing first: the resolved-ID columns, missing interval, metric naming, CLI version not keyed, model-aware temperature, env-list sync and dry-run.
• go build -tags gtk3 -o bin/KeyLint .: ok.
• cd frontend && npm test: 37 files, 496 tests pass.
• wire gen / bindings: not needed.

Smoke run (made before the review fixes; nothing re-run since)

One sample per suite, filtered with go test -run. 6 model calls, plus 5 non-model CLI invocations (--version ×4, auth status ×1).

Suite	Sample	Pipeline → resolved	Haiku 5.5	Sonnet 5.5
Fix	en-grammar-basics	haiku → claude-haiku-5-5	0.97	0.96
Pyramidize	email-meeting-request	sonnet → claude-sonnet-5-5	0.92	0.82

These are single samples, not measurements. The Fix smoke sample came from the holdout split; future smoke runs should use a tune sample, e.g. EVAL_SPLIT=tune go test -tags eval ./internal/features/enhance/ -run 'TestEvalFix/^interpunktion$' -v.

Not done

• No two-judge baseline and no bridge run.
• Per-form deterministic checks.
• The API judge route is still uncalled (no key).
• Never merged.