Skip to content

[T-041b] Strategy checkpoints (Memento), leader advisory lock, orchestrator restart-resume #86

Closed

Description

@CodewithJha

Task T-041b · Milestone M3 — Oracles + runner + store · Size S · Priority P0 · Epic #3

Implements NFR-REL-002

Branch feat/t-041b-strategy-checkpoints-memento-leader-advisory — one task = one branch = one PR (doc 32 §6)

Summary

Persist strategy.state_dict() to strategy_states every tick and restore with load_state_dict on start; guard each campaign with pg_try_advisory_lock(hashtext(campaign_id)); integration test that kills/restarts the orchestrator and resumes.

Context / motivation

NFR-REL-002: orchestrator restart resumes running campaigns from DB state; two orchestrators must never drive the same campaign.

Scope

In scope

  • runner/checkpoint.py, advisory-lock helper in the store adapter (store/locks.py).
  • Restart-resume integration test: next proposals after resume equal those of an uninterrupted run (deterministic strategy).

Implementation guidance

  • State must be JSON-serialisable (no pickles). Lock lives on a dedicated connection held for the loop lifetime.
  • Standards: respect the layering contracts (doc 16 §2) and conventions in doc 34; no hard-coded tunables — thresholds/defaults come from typed settings (doc 27); no magic numbers (ruff PLR2004); public APIs typed (mypy strict) and documented.

Acceptance criteria

  • Restart-resume test passes.
  • Second orchestrator cannot acquire the lock (test).
  • Tests listed below added and green in CI; coverage of new code meets the gate in doc 18 §3.
  • Docs, config reference and CHANGELOG Unreleased updated where behaviour, keys or interfaces changed.
  • Definition of Done (doc 32 §3) satisfied.

Tests required

  • tests/integration/runner/test_restart_resume.py, test_leader_lock.py.

Dependencies

Blocked by: #85 (T-041a)

Blocks: #91 (T-045b), #95 (T-048)

Parallelization notes

Files / paths owned by this task — avoid editing other paths; shared files (pyproject.toml entry points, cli/main.py, api/routers/__init__.py, router.tsx, docker/compose.yaml, ci.yml) are append-only:

  • backend/src/roboprobe/runner/checkpoint.py
  • backend/src/roboprobe/store/locks.py

After T-041a.

Definition of Done

Per doc 32 §3 and AGENTS.md §5: lint, format, mypy strict and import-linter green; tests added and passing; docs updated; PR template completed with Closes #<this issue>; Conventional Commit messages referencing T-041b (doc 16 §6); commits authored only by the repository owner identity (AGENTS.md §4); CI green; reviewer checklist (doc 32 §5.1) satisfied.

References

Design docs: doc 05 §5.1 · doc 34 §4 · doc 02 §5

Requirements: NFR-REL-002 (traceability: doc 28)

External:

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:runnerQueue, workers, campaign orchestrationmilestone:M3M3 — Oracles + runner + storepriority:P0Critical path / blocks otherssize:S<= 1 daytaskChild task of a milestone epictype:featureNew functionality

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions