Repository navigation
fix: stabilize model-override and independent behavior cases - #18
Merged
Merged
Conversation
model-override: the judge rubric asked for "fast ran" while the skill requires "the file does not show that any run took place". Rewrite the criteria as five numbered requirements with explicit not-a-failure and FAIL lists. 5 of 7 failed runs were rubric-side. independent: 2 of 2 failed runs on v0.9.0 were LLM graders skipped at the cost ceiling, counted as failed. Document the skip and a per-run budget for single-case reruns. Skill wording, for the 3 runs where the agent was at fault: do not advise changing what the host offers on the strength of a record; re-run only the one doubted claim of a read-only worker. On this tree, 6 runs per case with claude-sonnet-5-5: model-override 6 of 6, independent 6 of 6, $3.85. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Contributor
Author
|
Shipped in 0.10.0 through #17. The 0.9.1 entry and manifest bump on this branch were folded into 0.10.0; no v0.9.1 tag exists. The |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
On tag
v0.9.0, six runs per case gaveindependent4 of 6 andmodel-override3 of 6, against 3 of 3 in the 0.9.0 changelog.Cause
model-override(7 failed runs in 3 result files)independent(3 failed runs)On
v0.9.0alone: 0 agent faults, 3 rubric, 2 cost cap.model-override: the rubric asked for "fast ran" while the skill requires "the file does not show that any run took place". A blind regrade of all 15 reports with the same criteria passed 13; it failed only the 2 that tell the user to make premium available on the host. The result files hold judge votes only, so this attribution is inferred.independent: both failedv0.9.0runs have their LLM graders marked "skipped: cost ceiling". The eval counts those as failed and leavespartial: false.Change
evals/model-override/case.yaml: criteria as five numbered requirements, a not-a-failure list, an explicit FAIL list.skills/ai-native-sdlc/references/model-selection.md: one clause, no advice to change what the host offers on the strength of a record.skills/ai-native-sdlc/references/economy.md: one sentence, re-run only the one doubted claim of a read-only worker.docs/behavior-checks.md: cost-ceiling skip warning, $1 per run for single-case reruns.Verification
python scripts/test_behavior_check.py,scripts/test_skills.py,scripts/check_version.py: pass.claude plugin eval, Claude Code 2.1.319 in WSL2,claude-sonnet-5-5agent and judge, with-plugin arm, 6 runs per case:model-override6 of 6 ($0.58),independent6 of 6 ($3.27). No grader skipped.Not done
🤖 Generated with Claude Code