Repository navigation
Description
Task
T-052· Milestone M4 — Baselines + metrics + experiments · Size S · Priority P0 · Epic #4Implements FR-SEL-002 · FR-SEL-003
Branch
feat/t-052-analytic-benchmarks-and-benchmark-backend— one task = one branch = one PR (doc 32 §6)
Summary
Implement selection/benchmarks.py with the two functions of doc 07 §9 (known failure modes, ~0.7 % failure volume for multi_mode_6d) plus a BenchmarkBackend adapter so experiments can run analytic campaigns through the normal pipeline.
Context / motivation
Lets selection algorithms be tested statistically in milliseconds (doc 18 §4.4) and powers EXP-02.
Scope
In scope
- Functions with GT mode labelling helper
label_mode(u) -> {'A','B','C','D',None}. backends/benchmark/backend.pyproducing a synthetic trace whose oracle evaluation yields the benchmark ρ (or a direct-ρ shortcut flagged in capabilities).- Benchmark spaces
configs/scenario_spaces/branin_2d.yaml,multi_mode_6d.yaml.
Implementation guidance
- Seeded noise from the run stream. Validate failure volume by QMC (≈ 0.7 % ± tolerance) in a test.
- Standards: respect the layering contracts (doc 16 §2) and conventions in doc 34; no hard-coded tunables — thresholds/defaults come from typed settings (doc 27); no magic numbers (ruff
PLR2004); public APIs typed (mypy strict) and documented.
Acceptance criteria
- Volume check ≈ 0.7 % (documented tolerance).
- Each mode's canonical point fails; nominal passes.
- Tests listed below added and green in CI; coverage of new code meets the gate in doc 18 §3.
- Docs, config reference and CHANGELOG
Unreleasedupdated where behaviour, keys or interfaces changed. - Definition of Done (doc 32 §3) satisfied.
Tests required
tests/unit/selection/test_benchmarks.py.
Dependencies
Blocked by: #84 (T-050a), #48 (T-035a)
Blocks: #115 (T-057), #116 (T-058), #119 (T-059c)
Parallelization notes
Files / paths owned by this task — avoid editing other paths; shared files (pyproject.toml entry points, cli/main.py, api/routers/__init__.py, router.tsx, docker/compose.yaml, ci.yml) are append-only:
backend/src/roboprobe/selection/benchmarks.pybackend/src/roboprobe/backends/benchmark/backend.pyconfigs/scenario_spaces/branin_2d.yamlconfigs/scenario_spaces/multi_mode_6d.yaml
Independent.
Definition of Done
Per doc 32 §3 and AGENTS.md §5: lint, format, mypy strict and import-linter green; tests added and passing; docs updated; PR template completed with Closes #<this issue>; Conventional Commit messages referencing T-052 (doc 16 §6); commits authored only by the repository owner identity (AGENTS.md §4); CI green; reviewer checklist (doc 32 §5.1) satisfied.
References
Design docs: doc 07 §9 · doc 18 §4.4 · doc 19 §4 · doc 10 §7.2
Requirements: FR-SEL-002 · FR-SEL-003 (traceability: doc 28)
Metadata
Metadata
Assignees
Labels
Projects
- StatusShow more project fieldsBacklog
Activity