Skip to content

[T-100a] Metrics: MET-01 (discovery rate), MET-03 (first failure, censored), MET-05 (cost per mode), MET-06 (reproducibility), MET-07 (robustness Δ) #104

Closed

Description

@CodewithJha

Task T-100a · Milestone M4 — Baselines + metrics + experiments · Size S · Priority P0 · Epic #4

Implements FR-EXP-001

Branch feat/t-100a-metrics-met-01-discovery-rate — one task = one branch = one PR (doc 32 §6)

Summary

Implement per-campaign counting metrics in metrics/definitions.py (or metrics/campaign_metrics.py) with exact formulas of doc 19 §3.

Context / motivation

Metric bugs silently invalidate the research claims; formulas are pre-registered in doc 19.

Scope

In scope

  • Pure functions over RunRecord sequences; MET-03 right-censored at B (returns value + censored flag).
  • MET-07 with bootstrap CI hook (statistics from T-101a).

Implementation guidance

  • Exclude confirm-tagged runs from strategy comparison metrics (doc 09 §6). Document every formula in the docstring with the doc reference.
  • Standards: respect the layering contracts (doc 16 §2) and conventions in doc 34; no hard-coded tunables — thresholds/defaults come from typed settings (doc 27); no magic numbers (ruff PLR2004); public APIs typed (mypy strict) and documented.

Acceptance criteria

  • Hand-computed examples for each metric (≥ 3 cases each incl. edge cases: no failures, all INVALID).
  • ≥ 90 % coverage.
  • Tests listed below added and green in CI; coverage of new code meets the gate in doc 18 §3.
  • Docs, config reference and CHANGELOG Unreleased updated where behaviour, keys or interfaces changed.
  • Definition of Done (doc 32 §3) satisfied.

Tests required

  • tests/unit/metrics/test_campaign_metrics.py.

Dependencies

Blocked by: #61 (T-011a)

Blocks: #109 (T-102b)

Parallelization notes

Files / paths owned by this task — avoid editing other paths; shared files (pyproject.toml entry points, cli/main.py, api/routers/__init__.py, router.tsx, docker/compose.yaml, ci.yml) are append-only:

  • backend/src/roboprobe/metrics/campaign_metrics.py

Parallel with T-100b (separate module).

Definition of Done

Per doc 32 §3 and AGENTS.md §5: lint, format, mypy strict and import-linter green; tests added and passing; docs updated; PR template completed with Closes #<this issue>; Conventional Commit messages referencing T-100a (doc 16 §6); commits authored only by the repository owner identity (AGENTS.md §4); CI green; reviewer checklist (doc 32 §5.1) satisfied.

References

Design docs: doc 19 §3 · doc 09 §6 · doc 02 §3.7

Requirements: FR-EXP-001 (traceability: doc 28)

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:metricsEvaluation metrics MET-xxmilestone:M4M4 — Baselines + metrics + experimentspriority:P0Critical path / blocks otherssize:S<= 1 daytaskChild task of a milestone epictype:featureNew functionality

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions