Skip to content

[T-105a] Run EXP-02 (analytic) and EXP-07 (dimension/noise scalability); evidence report #207

Open

Description

@CodewithJha

Task T-105a · Milestone M10 — Evaluation · Size S · Priority P0 · Epic #10

Implements FR-EXP-003

Branch spike/t-105a-run-exp-02-analytic-and — one task = one branch = one PR (doc 32 §6)

Summary

Execute EXP-02 and EXP-07, generate experiment reports, and write docs/evidence/EXP-02.md / EXP-07.md with environment manifest, raw-data Parquet hash and honest interpretation.

Context / motivation

Cheap experiments first validate the pipeline before the 48 000-run EXP-01.

Scope

In scope

  • Reports committed as HTML artefacts in a release asset or docs/evidence/ (size-limited).

Implementation guidance

  • Record commands verbatim in the evidence (roboprobe experiment run configs/experiments/exp02_analytic.yaml, roboprobe experiment report <id> -o …); same for EXP-07.
  • Create docs/evidence/_experiment_template.md and use it: Purpose/hypotheses · Config hash · Environment manifest (CPU, RAM, OS, versions) · Compute time · Results table (median/IQR per metric/strategy, p, p_Holm, Â12) · Figures (exported SVG) · Interpretation · Deviations & threats · Raw data (Parquet path + SHA-256, release-asset link).
  • Large HTML reports and Parquet files (> 5 MB) go to a GitHub Release asset created by the owner instead of git.
  • Standards: respect the layering contracts (doc 16 §2) and conventions in doc 34; no hard-coded tunables — thresholds/defaults come from typed settings (doc 27); no magic numbers (ruff PLR2004); public APIs typed (mypy strict) and documented.

Acceptance criteria

  • Reports + evidence docs committed; hashes recorded.
  • Template created and used.
  • Any deviation from pre-registration explicitly listed.
  • Tests listed below added and green in CI; coverage of new code meets the gate in doc 18 §3.
  • Docs, config reference and CHANGELOG Unreleased updated where behaviour, keys or interfaces changed.
  • Definition of Done (doc 32 §3) satisfied.

Tests required

  • N/A (evaluation) — reproducible command lines included.

Dependencies

Blocked by: #206 (T-104), #141 (T-079)

Blocks: #210 (T-120a)

Parallelization notes

Files / paths owned by this task — avoid editing other paths; shared files (pyproject.toml entry points, cli/main.py, api/routers/__init__.py, router.tsx, docker/compose.yaml, ci.yml) are append-only:

  • docs/evidence/EXP-02.md
  • docs/evidence/EXP-07.md

Parallel with T-105b compute (different machines/time).

Definition of Done

Per doc 32 §3 and AGENTS.md §5: lint, format, mypy strict and import-linter green; tests added and passing; docs updated; PR template completed with Closes #<this issue>; Conventional Commit messages referencing T-105a (doc 16 §6); commits authored only by the repository owner identity (AGENTS.md §4); CI green; reviewer checklist (doc 32 §5.1) satisfied.

References

Design docs: doc 19 §4 · doc 19 §7 · doc 19 §8

Requirements: FR-EXP-003 (traceability: doc 28)

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:evalEvaluation campaigns and success criteriaarea:experimentsExperiment runner and EXP-xx protocolsmilestone:M10M10 — Evaluationpriority:P0Critical path / blocks otherssize:S<= 1 daytaskChild task of a milestone epictype:researchTime-boxed spike / investigation

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions