Skip to content

[T-064] Weekly benchmark-full.yml workflow (EXP-02 + reduced EXP-01) with regression comment #123

Closed

Description

@CodewithJha

Task T-064 · Milestone M5 — Intelligent selection · Size S · Priority P2 · Epic #5

Implements FR-EXP-001

Branch ci/t-064-weekly-workflow-exp-02-reduced — one task = one branch = one PR (doc 32 §6)

Summary

Add the weekly/manual workflow of doc 20 §1 running EXP-02 analytic and a reduced EXP-01 (10 seeds), comparing with a baseline JSON and posting a summary (job summary; PR comment when manually triggered on a PR).

Context / motivation

Detects slow scientific drift (e.g. S-PROBE getting worse) that PR smoke gates miss.

Scope

In scope

  • .github/workflows/benchmark-full.yml; scripts/compare_benchmark.py.

Implementation guidance

  • ≤ 45 min on hosted runners; cache the uv env.
  • Standards: respect the layering contracts (doc 16 §2) and conventions in doc 34; no hard-coded tunables — thresholds/defaults come from typed settings (doc 27); no magic numbers (ruff PLR2004); public APIs typed (mypy strict) and documented.

Acceptance criteria

  • Workflow runs on workflow_dispatch and schedule; regression beyond tolerance fails.
  • Tests listed below added and green in CI; coverage of new code meets the gate in doc 18 §3.
  • Docs, config reference and CHANGELOG Unreleased updated where behaviour, keys or interfaces changed.
  • Definition of Done (doc 32 §3) satisfied.

Tests required

  • Workflow run.

Dependencies

Blocked by: #122 (T-061), #109 (T-102b)

Blocks: no other task (leaf)

Parallelization notes

Files / paths owned by this task — avoid editing other paths; shared files (pyproject.toml entry points, cli/main.py, api/routers/__init__.py, router.tsx, docker/compose.yaml, ci.yml) are append-only:

  • .github/workflows/benchmark-full.yml
  • scripts/compare_benchmark.py

CI only.

Definition of Done

Per doc 32 §3 and AGENTS.md §5: lint, format, mypy strict and import-linter green; tests added and passing; docs updated; PR template completed with Closes #<this issue>; Conventional Commit messages referencing T-064 (doc 16 §6); commits authored only by the repository owner identity (AGENTS.md §4); CI green; reviewer checklist (doc 32 §5.1) satisfied.

References

Design docs: doc 20 §1 · doc 19 §4

Requirements: FR-EXP-001 (traceability: doc 28)

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:ciCI/CD workflowsarea:evalEvaluation campaigns and success criteriamilestone:M5M5 — Intelligent selectionpriority:P2Important, not blockingsize:S<= 1 daytaskChild task of a milestone epictype:infraCI, build, deployment infrastructure

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions