Repository navigation
Description
Task
T-105a· Milestone M10 — Evaluation · Size S · Priority P0 · Epic #10Implements FR-EXP-003
Branch
spike/t-105a-run-exp-02-analytic-and— one task = one branch = one PR (doc 32 §6)
Summary
Execute EXP-02 and EXP-07, generate experiment reports, and write docs/evidence/EXP-02.md / EXP-07.md with environment manifest, raw-data Parquet hash and honest interpretation.
Context / motivation
Cheap experiments first validate the pipeline before the 48 000-run EXP-01.
Scope
In scope
- Reports committed as HTML artefacts in a release asset or
docs/evidence/(size-limited).
Implementation guidance
- Record commands verbatim in the evidence (
roboprobe experiment run configs/experiments/exp02_analytic.yaml,roboprobe experiment report <id> -o …); same for EXP-07. - Create
docs/evidence/_experiment_template.mdand use it: Purpose/hypotheses · Config hash · Environment manifest (CPU, RAM, OS, versions) · Compute time · Results table (median/IQR per metric/strategy, p, p_Holm, Â12) · Figures (exported SVG) · Interpretation · Deviations & threats · Raw data (Parquet path + SHA-256, release-asset link). - Large HTML reports and Parquet files (> 5 MB) go to a GitHub Release asset created by the owner instead of git.
- Standards: respect the layering contracts (doc 16 §2) and conventions in doc 34; no hard-coded tunables — thresholds/defaults come from typed settings (doc 27); no magic numbers (ruff
PLR2004); public APIs typed (mypy strict) and documented.
Acceptance criteria
- Reports + evidence docs committed; hashes recorded.
- Template created and used.
- Any deviation from pre-registration explicitly listed.
- Tests listed below added and green in CI; coverage of new code meets the gate in doc 18 §3.
- Docs, config reference and CHANGELOG
Unreleasedupdated where behaviour, keys or interfaces changed. - Definition of Done (doc 32 §3) satisfied.
Tests required
- N/A (evaluation) — reproducible command lines included.
Dependencies
Blocked by: #206 (T-104), #141 (T-079)
Blocks: #210 (T-120a)
Parallelization notes
Files / paths owned by this task — avoid editing other paths; shared files (pyproject.toml entry points, cli/main.py, api/routers/__init__.py, router.tsx, docker/compose.yaml, ci.yml) are append-only:
docs/evidence/EXP-02.mddocs/evidence/EXP-07.md
Parallel with T-105b compute (different machines/time).
Definition of Done
Per doc 32 §3 and AGENTS.md §5: lint, format, mypy strict and import-linter green; tests added and passing; docs updated; PR template completed with Closes #<this issue>; Conventional Commit messages referencing T-105a (doc 16 §6); commits authored only by the repository owner identity (AGENTS.md §4); CI green; reviewer checklist (doc 32 §5.1) satisfied.
References
Design docs: doc 19 §4 · doc 19 §7 · doc 19 §8
Requirements: FR-EXP-003 (traceability: doc 28)
Metadata
Metadata
Assignees
Labels
Projects
- StatusShow more project fieldsBacklog
Activity
configs/experiments/*.yaml) validated in CI #206fix(experiments): enable analytical mode evidence in the CLI (T-105a) (…