Skip to content

[T-039b] roboprobe oracle reevaluate <run_id>: offline re-judgement from stored traces #93

Closed

Description

@CodewithJha

Task T-039b · Milestone M3 — Oracles + runner + store · Size XS · Priority P1 · Epic #3

Implements FR-ORC-006

Branch feat/t-039b-offline-re-judgement-from-stored — one task = one branch = one PR (doc 32 §6)

Summary

CLI + service that loads a run's stored trace and task, re-runs the VerdictEngine (optionally with new thresholds) and reports differences against the stored verdict.

Context / motivation

FR-ORC-006 AC: reevaluation reproduces stored verdicts; also used after threshold changes (doc 27 §6 rule 5).

Scope

In scope

  • oracles/reevaluate.py application helper; cli/oracle.py.
  • --thresholds file.toml override; --write to persist new results (with oracle_set_version).

Implementation guidance

  • Uses ReplayBackend-free path: read trace via trace_io, no simulator.
  • Standards: respect the layering contracts (doc 16 §2) and conventions in doc 34; no hard-coded tunables — thresholds/defaults come from typed settings (doc 27); no magic numbers (ruff PLR2004); public APIs typed (mypy strict) and documented.

Acceptance criteria

  • Re-evaluating unchanged runs yields identical verdict/ρ.
  • Override thresholds changes results only where expected.
  • Tests listed below added and green in CI; coverage of new code meets the gate in doc 18 §3.
  • Docs, config reference and CHANGELOG Unreleased updated where behaviour, keys or interfaces changed.
  • Definition of Done (doc 32 §3) satisfied.

Tests required

  • tests/integration/oracles/test_reevaluate.py.

Dependencies

Blocked by: #77 (T-039a), #66 (T-013a), #61 (T-011a), #20 (T-008)

Blocks: no other task (leaf)

Parallelization notes

Files / paths owned by this task — avoid editing other paths; shared files (pyproject.toml entry points, cli/main.py, api/routers/__init__.py, router.tsx, docker/compose.yaml, ci.yml) are append-only:

  • backend/src/roboprobe/oracles/reevaluate.py
  • backend/src/roboprobe/cli/oracle.py

Independent.

Definition of Done

Per doc 32 §3 and AGENTS.md §5: lint, format, mypy strict and import-linter green; tests added and passing; docs updated; PR template completed with Closes #<this issue>; Conventional Commit messages referencing T-039b (doc 16 §6); commits authored only by the repository owner identity (AGENTS.md §4); CI green; reviewer checklist (doc 32 §5.1) satisfied.

References

Design docs: doc 08 §1 · doc 02 §3.4 · doc 27 §6

Requirements: FR-ORC-006 (traceability: doc 28)

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:cliCommand-line interfacearea:oracleVerdict oracles / safety propertiesgood first issueGood for newcomersmilestone:M3M3 — Oracles + runner + storepriority:P1Required for milestone exitsize:XS< 2 hourstaskChild task of a milestone epictype:featureNew functionality

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions