Repository navigation
fix(llm): isolate Claude Code CLI calls from the user's sessions - #127
Merged
Merged
Conversation
--setting-sources "" does not stop Claude Code's auto-memory: with KeyLint's working directory inherited, the memory file of the matching ~/.claude/projects/<slug> was injected into every Fix and Pyramidize call (measured 2026-10-04, while building the CLI eval in #105). - Every spawn, the status probes included, runs in an empty KeyLint-owned directory under the user cache dir. Without a cache dir it uses a private os.MkdirTemp directory for the process (random name, 0700) instead of a shared fixed path; if that fails too the call fails with a clear error rather than inheriting the cwd. - CLAUDE_CODE_DISABLE_AUTO_MEMORY=1 is set; an inherited value cannot turn it back on. - Inherited alias-remapping (ANTHROPIC_DEFAULT_*_MODEL, ANTHROPIC_SMALL_FAST_MODEL), thinking-budget and output-cap variables are stripped like the credential ones. - Tests point the cache dir at a temp dir on every OS. Co-Authored-By: Claude <noreply@anthropic.com>
0xMMA
added a commit
that referenced
this pull request
Oct 11, 2026
Review follow-ups on the two-judge instrument: - Per-judge columns are keyed by the ID that answered, as the configKey is. Keyed by the requested name, a baseline judged as "haiku" and a run judged as "claude-haiku-5-5" shared a key but no column, and a drop from 0.95 to 0.40 read "inconclusive", exit 0. A judge with no interval on one side is now "not comparable", never "overlap". - SupportsTemperature is model-aware: on the API, Claude models from the 4.7 generation on take no temperature, so the 5.5 judges record null with a note instead of sending 0 (a 400). The Sonnet 4.5 bridge judge keeps temperature 0. - claudeCodeVersion leaves the configKey (the CLI updates itself); the baseline lists every version seen and --compare reports a change. The output names the metrics that flagged a verdict (regressedMetrics, improvedMetrics), and only for a comparison actually made. - eval.sh: its env -u list is tested against blockedCLIEnv; --dry-run asks the CLI nothing, --preflight adds the sign-in check; stale comments fixed. - Docs: API-era baselines are not comparable with this instrument on any route; the 2026-10-04 CLI baselines are marked as possibly affected by the auto-memory leak (#127) and superseded, and the readings built on them are flagged unconfirmed. Co-Authored-By: Claude <noreply@anthropic.com>
0xMMA
added a commit
that referenced
this pull request
Oct 11, 2026
Review follow-ups on the two-judge instrument: - Per-judge columns are keyed by the ID that answered, as the configKey is. Keyed by the requested name, a baseline judged as "haiku" and a run judged as "claude-haiku-5-5" shared a key but no column, and a drop from 0.95 to 0.40 read "inconclusive", exit 0. A judge with no interval on one side is now "not comparable", never "overlap". - SupportsTemperature is model-aware: on the API, Claude models from the 4.7 generation on take no temperature, so the 5.5 judges record null with a note instead of sending 0 (a 400). The Sonnet 4.5 bridge judge keeps temperature 0. - claudeCodeVersion leaves the configKey (the CLI updates itself); the baseline lists every version seen and --compare reports a change. The output names the metrics that flagged a verdict (regressedMetrics, improvedMetrics), and only for a comparison actually made. - eval.sh: its env -u list is tested against blockedCLIEnv; --dry-run asks the CLI nothing, --preflight adds the sign-in check; stale comments fixed. - Docs: API-era baselines are not comparable with this instrument on any route; the 2026-10-04 CLI baselines are marked as possibly affected by the auto-memory leak (#127) and superseded, and the readings built on them are flagged unconfirmed. Co-Authored-By: Claude <noreply@anthropic.com>
0xMMA
added a commit
that referenced
this pull request
Oct 11, 2026
…105) * chore(eval): run both eval suites through the Claude Code CLI PR #105 rebased onto main as one commit (the original eleven are on backup/eval-via-claude-code-pre-rebase). Pipeline and judge run through the installed Claude Code CLI by default; the judge records temperature null plus a note; claudeCodeVersion is recorded; .env keys load only when an API provider is in the run. Claude Code CLI calls run in an empty KeyLint-owned directory with auto-memory off, and inherited thinking, output-cap and alias-remapping variables are stripped. Resolved against main: - Resolved model IDs: main's Response.Model, answeringModel and llm.ModelRecorder replace the PR's ResolvedModels; summary.json keeps main's shape (model = resolved ID, modelRequested). The judge records its own resolvedModel through a second recorder. UsesAPIKey and SupportsTemperature move to internal/llm/traits.go. - Thinking: main's Request.NoThinking replaces the PR's DisableThinking. The CLI keeps ignoring it, as on main, so EVAL_THINKING / --thinking are gone; a Fix run records what the service does (fixThinkingLabel). - Judge: main's noEffort and the PR's nil temperature and deadline. - Effort, model families, typed errors and the rest of main unchanged. Co-Authored-By: Claude <noreply@anthropic.com> * feat(eval): score every sample with two judges Haiku 5.5 and Sonnet 5.5 judge each sample in both suites, both through the Claude Code CLI by default (decision 2026-10-11, E3 phase 1). Full model IDs: neither has a dated snapshot, and each judge records the ID that answered. - summary.json: judges[] (with resolvedModel), avgJudges, judgeCounts; results.jsonl: judges / judgeErrors keyed by judge model. - eval-aggregate.sh: one interval per judge (run and sample level), per-judge coverage; nothing averages across judges. Runs from before the set (judge / avgJudge) read as a set of one. - configKey: the judge provider, resolved model and temperature fields hold one "+"-joined entry per judge, ordered by ID. A single-judge key is unchanged, so every recorded baseline reads "not comparable" against a two-judge run instead of crashing. - --compare: interval rule per metric (deterministic, each judge); any metric clearly worse is a regression, an improvement needs one clearly better and none worse. - EVAL_JUDGES=a,b,c chooses the set (bridge run with the old Sonnet 4.5 judge); EVAL_JUDGE_MODEL is a set of one; both, a hole or a duplicate are refused. eval.sh shows the set; go test timeout 10800 s. Co-Authored-By: Claude <noreply@anthropic.com> * docs(eval): two judges in CLAUDE.md and both quality-status files Instrument note only, no new numbers: the single-judge baselines read "not comparable" against a two-judge run until new ones are recorded. Co-Authored-By: Claude <noreply@anthropic.com> * fix(eval): judge columns by resolved ID; model-aware judge temperature Review follow-ups on the two-judge instrument: - Per-judge columns are keyed by the ID that answered, as the configKey is. Keyed by the requested name, a baseline judged as "haiku" and a run judged as "claude-haiku-5-5" shared a key but no column, and a drop from 0.95 to 0.40 read "inconclusive", exit 0. A judge with no interval on one side is now "not comparable", never "overlap". - SupportsTemperature is model-aware: on the API, Claude models from the 4.7 generation on take no temperature, so the 5.5 judges record null with a note instead of sending 0 (a 400). The Sonnet 4.5 bridge judge keeps temperature 0. - claudeCodeVersion leaves the configKey (the CLI updates itself); the baseline lists every version seen and --compare reports a change. The output names the metrics that flagged a verdict (regressedMetrics, improvedMetrics), and only for a comparison actually made. - eval.sh: its env -u list is tested against blockedCLIEnv; --dry-run asks the CLI nothing, --preflight adds the sign-in check; stale comments fixed. - Docs: API-era baselines are not comparable with this instrument on any route; the 2026-10-04 CLI baselines are marked as possibly affected by the auto-memory leak (#127) and superseded, and the readings built on them are flagged unconfirmed. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Michael <0xMMA@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The Claude Code CLI provider now runs every spawn isolated from the user's own Claude Code sessions. That covers completions as well as the
--version/auth statusprobes.%LocalAppData%\KeyLint\claude-cli,~/.cache/KeyLint/claude-cli, or~/Library/Caches/KeyLint/claude-cli.os.MkdirTempdirectory created for the process (random name, mode 0700).CLAUDE_CODE_DISABLE_AUTO_MEMORY=1is set on every spawn, and an inherited value cannot override it.ANTHROPIC_DEFAULT_{OPUS,SONNET,HAIKU}_MODEL,ANTHROPIC_SMALL_FAST_MODEL,MAX_THINKING_TOKENSandCLAUDE_CODE_MAX_OUTPUT_TOKENSare removed, the same way the credential variables already were. The log line now reads "environment variables removed".Why
--setting-sources ""does not stop Claude Code's auto-memory. With KeyLint's working directory inherited (the install folder, or a developer's checkout), the memory file of the matching~/.claude/projects/<slug>was injected into every Fix and Pyramidize call. This was measured on 2026-10-04 while building the CLI eval in #105.It is a privacy problem: the user's notes from their own Claude Code sessions went along with every grammar fix. It is also a correctness problem, because those notes can change the answer. Alias-remapping variables in the user's shell could likewise make the CLI answer with a model other than the one KeyLint asked for.
Shared fallback dropped: an earlier draft fell back to a fixed
os.TempDir()/keylint-claude-cli. That path is shared and predictable, so another account on the machine could create it first. The fallback is now a per-processos.MkdirTempdirectory. Erroring out was the alternative, but a missing cache dir is rare and the private directory is just as neutral. Only when no directory can be made at all does the call fail; the system-prompt temp file would fail in that situation anyway.Not verified
~/.claude/CLAUDE.md, or aCLAUDE.mdin a parent of the cache dir such as the home directory.CLAUDE_CODE_DISABLE_AUTO_MEMORYclose it.~/.claude, which this PR does not touch.How verified
internal/llm(stub CLI, no model calls):cmd.Diris dropped.TestMainpoints the cache dir at a temp dir on every OS, so no test writes to the real cache.go vet -tags gtk3 ./...,go test -tags gtk3 ./internal/...andgo build -tags gtk3 -o bin/KeyLint .all pass.wire gen/ bindings run.#105 (the CLI eval) is rebased on top of this branch and depends on it. Merge this first.