Skip to content

fix: stabilize model-override and independent behavior cases - #18

Merged
sofus-common merged 2 commits into
mainfrom
claude/focused-shirley-f818f2
Nov 15, 2026
Merged

sofus-common merged 2 commits into
mainfrom
claude/focused-shirley-f818f2

Conversation

@sofus-common

Copy link
Copy Markdown
Contributor

Why

On tag v0.9.0, six runs per case gave independent 4 of 6 and model-override 3 of 6, against 3 of 3 in the 0.9.0 changelog.

Cause

Case Agent wrong Rubric ambiguous Harness cost cap
model-override (7 failed runs in 3 result files) 2 5 0
independent (3 failed runs) 1 0 2

On v0.9.0 alone: 0 agent faults, 3 rubric, 2 cost cap.

  • model-override: the rubric asked for "fast ran" while the skill requires "the file does not show that any run took place". A blind regrade of all 15 reports with the same criteria passed 13; it failed only the 2 that tell the user to make premium available on the host. The result files hold judge votes only, so this attribution is inferred.
  • independent: both failed v0.9.0 runs have their LLM graders marked "skipped: cost ceiling". The eval counts those as failed and leaves partial: false.

Change

  • evals/model-override/case.yaml: criteria as five numbered requirements, a not-a-failure list, an explicit FAIL list.
  • skills/ai-native-sdlc/references/model-selection.md: one clause, no advice to change what the host offers on the strength of a record.
  • skills/ai-native-sdlc/references/economy.md: one sentence, re-run only the one doubted claim of a read-only worker.
  • docs/behavior-checks.md: cost-ceiling skip warning, $1 per run for single-case reruns.

Verification

  • python scripts/test_behavior_check.py, scripts/test_skills.py, scripts/check_version.py: pass.
  • claude plugin eval, Claude Code 2.1.319 in WSL2, claude-sonnet-5-5 agent and judge, with-plugin arm, 6 runs per case: model-override 6 of 6 ($0.58), independent 6 of 6 ($3.27). No grader skipped.

Not done

  • No version bump or changelog entry. Two skill reference files changed; repo convention asks for 0.9.1 and the full suite before the entry is dated.
  • The other 15 cases did not run against the two new skill sentences.

🤖 Generated with Claude Code

sofus-common and others added 2 commits November 15, 2026 20:31
model-override: the judge rubric asked for "fast ran" while the skill
requires "the file does not show that any run took place". Rewrite the
criteria as five numbered requirements with explicit not-a-failure and
FAIL lists. 5 of 7 failed runs were rubric-side.

independent: 2 of 2 failed runs on v0.9.0 were LLM graders skipped at
the cost ceiling, counted as failed. Document the skip and a per-run
budget for single-case reruns.

Skill wording, for the 3 runs where the agent was at fault: do not
advise changing what the host offers on the strength of a record; re-run
only the one doubted claim of a read-only worker.

On this tree, 6 runs per case with claude-sonnet-5-5: model-override
6 of 6, independent 6 of 6, $3.85.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@sofus-common
sofus-common merged commit 807457a into main Nov 15, 2026
2 checks passed
@sofus-common

Copy link
Copy Markdown
Contributor Author

Shipped in 0.10.0 through #17. The 0.9.1 entry and manifest bump on this branch were folded into 0.10.0; no v0.9.1 tag exists. The model-override judge criteria were reworded further there, and the case is still not stable (4 of 6 on the shipped wording); see the 0.10.0 changelog entry.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant