Repository navigation
Description
Task
T-061· Milestone M5 — Intelligent selection · Size S · Priority P0 · Epic #5Implements FR-SEL-003
Branch
spike/t-061-calibrate-benchmark-thresholds-and-freeze— one task = one branch = one PR (doc 32 §6)
Summary
Run the calibration on analytic benchmarks only (EXP-02 style, fixed seed list), set gate thresholds with margin in tests/benchmark/thresholds.yaml, freeze S-PROBE defaults in settings, and document in docs/evidence/T-061-calibration.md before EXP-01 (threat to validity, doc 19 §8).
Context / motivation
Hyper-parameters must not be tuned on the evaluation space — this is the internal-validity safeguard.
Scope
In scope
- Calibration script
scripts/calibrate_benchmarks.py. - Thresholds YAML is config, not code.
Implementation guidance
- Record seeds, versions and raw results; any later change is a logged deviation (
docs/evidence/deviations.md). - Standards: respect the layering contracts (doc 16 §2) and conventions in doc 34; no hard-coded tunables — thresholds/defaults come from typed settings (doc 27); no magic numbers (ruff
PLR2004); public APIs typed (mypy strict) and documented.
Acceptance criteria
- Thresholds file committed with documented margins.
- Evidence note committed; defaults frozen.
- Tests listed below added and green in CI; coverage of new code meets the gate in doc 18 §3.
- Docs, config reference and CHANGELOG
Unreleasedupdated where behaviour, keys or interfaces changed. - Definition of Done (doc 32 §3) satisfied.
Tests required
- Benchmark tests read thresholds from YAML.
Dependencies
Blocked by: #119 (T-059c), #115 (T-057), #116 (T-058)
Blocks: #123 (T-064), #206 (T-104)
Parallelization notes
Files / paths owned by this task — avoid editing other paths; shared files (pyproject.toml entry points, cli/main.py, api/routers/__init__.py, router.tsx, docker/compose.yaml, ci.yml) are append-only:
backend/tests/benchmark/thresholds.yamlscripts/calibrate_benchmarks.pydocs/evidence/T-061-calibration.md
After S-PROBE lands.
Definition of Done
Per doc 32 §3 and AGENTS.md §5: lint, format, mypy strict and import-linter green; tests added and passing; docs updated; PR template completed with Closes #<this issue>; Conventional Commit messages referencing T-061 (doc 16 §6); commits authored only by the repository owner identity (AGENTS.md §4); CI green; reviewer checklist (doc 32 §5.1) satisfied.
References
Design docs: doc 19 §8 · doc 18 §4.4 · doc 19 §5
Requirements: FR-SEL-003 (traceability: doc 28)
Metadata
Metadata
Assignees
Labels
Projects
- StatusShow more project fieldsBacklog
Activity
benchmark-full.ymlworkflow (EXP-02 + reduced EXP-01) with regression comment #123configs/experiments/*.yaml) validated in CI #206