Skip to content

[T-050b] Strategy contract test suite (every registered strategy) #100

Closed

Description

@CodewithJha

Task T-050b · Milestone M4 — Baselines + metrics + experiments · Size S · Priority P0 · Epic #4

Implements FR-SEL-001 · FR-SEL-005 · FR-SEL-006 · FR-SEL-007 · NFR-PERF-002

Branch test/t-050b-strategy-contract-test-suite-every — one task = one branch = one PR (doc 32 §6)

Summary

Implement doc 18 §4.3 strategy contract: proposals in bounds & feasible; deterministic with seed; no duplicates with pending; ignores INVALID; state_dict round-trip reproduces next proposals; propose(q) returns exactly q (fewer only if exhausted); time limit.

Context / motivation

The executable definition of 'a valid strategy' — every new strategy (including plugins like S-CEM) must pass it.

Scope

In scope

  • tests/contract/selection/test_strategy_contract.py parametrised over registry.available() and a d=2/d=6 benchmark space.
  • Uses analytic benchmark observations (T-052) once available; synthetic random observations before.

Implementation guidance

  • Keep per-strategy runtime small (budget ≤ 30 in contract tests); heavy statistical checks live in benchmark tests.
  • Standards: respect the layering contracts (doc 16 §2) and conventions in doc 34; no hard-coded tunables — thresholds/defaults come from typed settings (doc 27); no magic numbers (ruff PLR2004); public APIs typed (mypy strict) and documented.

Acceptance criteria

  • Suite green for all registered strategies.
  • Documented in CONTRIBUTING (how to run for a plugin).
  • Tests listed below added and green in CI; coverage of new code meets the gate in doc 18 §3.
  • Docs, config reference and CHANGELOG Unreleased updated where behaviour, keys or interfaces changed.
  • Definition of Done (doc 32 §3) satisfied.

Tests required

  • The suite.

Dependencies

Blocked by: #84 (T-050a)

Blocks: #124 (T-062)

Parallelization notes

Files / paths owned by this task — avoid editing other paths; shared files (pyproject.toml entry points, cli/main.py, api/routers/__init__.py, router.tsx, docker/compose.yaml, ci.yml) are append-only:

  • backend/tests/contract/selection/**

Test-only.

Definition of Done

Per doc 32 §3 and AGENTS.md §5: lint, format, mypy strict and import-linter green; tests added and passing; docs updated; PR template completed with Closes #<this issue>; Conventional Commit messages referencing T-050b (doc 16 §6); commits authored only by the repository owner identity (AGENTS.md §4); CI green; reviewer checklist (doc 32 §5.1) satisfied.

References

Design docs: doc 18 §4.3 · doc 05 §4

Requirements: FR-SEL-001 · FR-SEL-005 · FR-SEL-006 · FR-SEL-007 · NFR-PERF-002 (traceability: doc 28)

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:selectionScenario selection strategies (BO, LSE, PROBE, baselines)milestone:M4M4 — Baselines + metrics + experimentspriority:P0Critical path / blocks otherssize:S<= 1 daytaskChild task of a milestone epictype:testTest suites and test infrastructure

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions