Skip to content

[T-120a] Run EXP-04 on Gazebo (RANDOM vs PROBE, 5 seeds × 80) and capture SC-4 interaction failure with video/rosbag #210

Open

Description

@CodewithJha

Task T-120a · Milestone M10 — Evaluation · Size M · Priority P0 · Epic #10

Implements FR-EXP-003 · FR-SIM-003

Branch spike/t-120a-run-exp-04-on-gazebo — one task = one branch = one PR (doc 32 §6)

Summary

Execute EXP-04 over nights, analyse, confirm at least one interaction failure (×3 confirmation), and record a demo video (gz GUI replay) and MCAP bag; write docs/evidence/EXP-04.md.

Context / motivation

SC-4 real-stack demonstration.

Scope

In scope

  • Video stored as release asset (not in git) with link.

Implementation guidance

  • Pre-flight: Gazebo nightly green (T-098); RTF and episode times from T-090/T-098 inform the schedule (≈ 22 h compute spread over nights, doc 19 §4).
  • Run exp04_gazebo.yaml (RANDOM vs PROBE, 5 seeds × 80; OFAT once × 30); monitor in the dashboard; follow RB-02 (slow RTF), RB-03 (Nav2 not ACTIVE), RB-04 (orphans).
  • Analyse modes, confirm interactions on the top modes (:confirm-interaction) and pick one clear interaction failure for SC-4.
  • Demo: replay the chosen failure with the Gazebo GUI (doc 17 §6) and screen-record ≤ 2 min; bag + video as release assets, linked from the evidence and README.
  • Standards: respect the layering contracts (doc 16 §2) and conventions in doc 34; no hard-coded tunables — thresholds/defaults come from typed settings (doc 27); no magic numbers (ruff PLR2004); public APIs typed (mypy strict) and documented.

Acceptance criteria

  • Evidence doc with interaction-failure case, replay links and video link.
  • At least one CONFIRMED interaction failure on Gazebo, or an honest negative result with analysis.
  • Video and bag links valid.
  • Tests listed below added and green in CI; coverage of new code meets the gate in doc 18 §3.
  • Docs, config reference and CHANGELOG Unreleased updated where behaviour, keys or interfaces changed.
  • Definition of Done (doc 32 §3) satisfied.

Tests required

  • N/A.

Dependencies

Blocked by: #205 (T-098), #207 (T-105a)

Blocks: #211 (T-120b), #212 (T-120c), #222 (T-121)

Parallelization notes

Files / paths owned by this task — avoid editing other paths; shared files (pyproject.toml entry points, cli/main.py, api/routers/__init__.py, router.tsx, docker/compose.yaml, ci.yml) are append-only:

  • docs/evidence/EXP-04.md

Long-running; parallel with T-108, T-125.

Definition of Done

Per doc 32 §3 and AGENTS.md §5: lint, format, mypy strict and import-linter green; tests added and passing; docs updated; PR template completed with Closes #<this issue>; Conventional Commit messages referencing T-120a (doc 16 §6); commits authored only by the repository owner identity (AGENTS.md §4); CI green; reviewer checklist (doc 32 §5.1) satisfied.

References

Design docs: doc 19 §4 · doc 01 §7 · doc 30 §RB-02 · doc 10 §6 · doc 17 §6

Requirements: FR-EXP-003 · FR-SIM-003 (traceability: doc 28)

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:evalEvaluation campaigns and success criteriaarea:simulatorSimulator backends (kinematic2d, Gazebo adapter)milestone:M10M10 — Evaluationpriority:P0Critical path / blocks otherssize:M1-3 daystaskChild task of a milestone epictype:researchTime-boxed spike / investigation

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions