Repository navigation
Description
Task
T-039a· Milestone M3 — Oracles + runner + store · Size S · Priority P0 · Epic #3Implements FR-ORC-003 · FR-ORC-004 · FR-ORC-006
Branch
feat/t-039a-verdictengine-invalid-fail-pass-min— one task = one branch = one PR (doc 32 §6)
Summary
Implement oracles/verdict.py (VerdictEngine.judge(result, task) -> RunOutcome) per doc 08 §5 and compute_run_metrics(trace, task) per doc 08 §6.
Context / motivation
This is where the robustness signal that drives learning (ρ) is produced; its invariant FAIL ⇔ ρ < 0 is the backbone of the selector.
Scope
In scope
- InfraCheck first → INVALID with
ρ = None. - FAIL if any failure-severity oracle violated; primary = earliest
first_violation_t; tiesCOL > INST > LOC > PLAN > STUCK > TIMEOUT; secondary violations kept. ρ = min_k r_kover failure-severity oracles; warnings (F-NEAR, F-OSC) attached.- Metrics:
duration_s, path_length_m, mean_speed, max_speed, min_clearance_m, time_below_d_safe_s, max_loc_error_m, mean_loc_error_m, n_recoveries, n_replans, goal_reached, t_goal_s, energy_proxy, rtf_mean. oracle_set_versionconstant bumped when any oracle changes (doc 33 §6).
Implementation guidance
- The engine receives oracles via constructor (built by the registry in bootstrap). Keep it pure.
- Standards: respect the layering contracts (doc 16 §2) and conventions in doc 34; no hard-coded tunables — thresholds/defaults come from typed settings (doc 27); no magic numbers (ruff
PLR2004); public APIs typed (mypy strict) and documented.
Acceptance criteria
- Invariant test: for every golden trace and 1 000 Hypothesis-generated traces,
verdict == FAIL ⇔ ρ < 0for valid runs. - Tie-break order tested.
- Metrics match hand-computed values on golden traces.
- Tests listed below added and green in CI; coverage of new code meets the gate in doc 18 §3.
- Docs, config reference and CHANGELOG
Unreleasedupdated where behaviour, keys or interfaces changed. - Definition of Done (doc 32 §3) satisfied.
Tests required
tests/unit/oracles/test_verdict.py,test_run_metrics.py,tests/property/oracles/test_verdict_invariant.py.
Dependencies
Blocked by: #70 (T-036b), #71 (T-036c), #72 (T-037a), #73 (T-037b), #74 (T-037c), #75 (T-038a), #76 (T-038b)
Blocks: #78 (T-034), #80 (T-040a), #93 (T-039b), #218 (T-131)
Parallelization notes
Files / paths owned by this task — avoid editing other paths; shared files (pyproject.toml entry points, cli/main.py, api/routers/__init__.py, router.tsx, docker/compose.yaml, ci.yml) are append-only:
backend/src/roboprobe/oracles/verdict.pybackend/src/roboprobe/oracles/metrics.py
Integration point of the oracle stream.
Definition of Done
Per doc 32 §3 and AGENTS.md §5: lint, format, mypy strict and import-linter green; tests added and passing; docs updated; PR template completed with Closes #<this issue>; Conventional Commit messages referencing T-039a (doc 16 §6); commits authored only by the repository owner identity (AGENTS.md §4); CI green; reviewer checklist (doc 32 §5.1) satisfied.
References
Design docs: doc 08 §5 · doc 08 §6 · doc 05 §6 · doc 02 §3.4 · doc 33 §6
Requirements: FR-ORC-003 · FR-ORC-004 · FR-ORC-006 (traceability: doc 28)
Metadata
Metadata
Assignees
Labels
Projects
- StatusShow more project fieldsBacklog
Activity
oracles/clearance.py) #70oracles/goal.py) #71oracles/progress.py) #72oracles/localization.py) #73oracles/nav.py) #74oracles/stability.py) #75oracles/oscillation.py) #76roboprobe oracle reevaluate <run_id>: offline re-judgement from stored traces #93