Repository navigation
Slurm executor: frozen-commit submission, import-origin check, node packing, dependency chaining #174
Description
Several projects running trade-study-style replicate loops (simulation-based calibration, design recovery, scenario grids) on Slurm clusters have each written the same submission layer, and found the same failure modes:
- Jobs read their code when they start, not when they are submitted. A commit, pull or branch switch in the submitting checkout changes what queued jobs run. The fix: submit from a detached worktree of one commit, pin it (
EXPECT_SHA), and have each job verify the commit and refuse a modified tracked tree. - Editable installs from another checkout win silently. Each job should verify that the package imports from its own worktree.
- Per-core partitions can carry a low account-wide CPU cap. Many small tasks are better packed onto whole nodes: run N tasks at a time on a node, each with cores/N CPUs, with per-task logs, failing the job if any task fails.
- Dependency chaining: later stages wait on earlier ones still queued or running.
sbatchrejectsafterokon jobs that already finished, so those must be checked as COMPLETED and skipped. - Resubmission reruns only missing units (existing outputs skipped, but only on a provenance match).
Proposal: an optional executor for run_grid / run_adaptive that materialises tasks as packed Slurm jobs with these guarantees, with the existing journal/checkpoint for resume. Prototypes exist as shell scripts (a frozen-worktree helper, a git/import guard, a node packer, and dependency-aware submission) and can be generalised.
Activity
jc-macdonald commentedon Oct 14, 2026
One more requirement from measurement: the executor should size tasks from a measured footprint, not from a guess. In one project, JAX CPU processes used about 2 cores and under 2 GB whatever they were allocated; tasks requested at 8–16 CPUs ran whole 48-core nodes at 2–19% efficiency. At 2 CPUs per process, the same nodes did 3× more work per core.
Suggested executor features:
seff/sacct(CPU efficiency, MaxRSS);jc-macdonald commentedon Oct 16, 2026
A failure mode from ACCIDDA/FLU_USA 13 (solver benchmark on Rockfish
parallelnodes) for the node-packing part of this issue.Invalid packing baseline from a fixed start time. The harness started P worker processes and told them to begin timed runs at a clock time
now + 30 s + 2 s * P. Setup (a reference solve and, for the 51-state JAX system, a 27 s compile) outlasted that window, so the single-process baseline included set-up and the apparent parallel efficiency was impossible (51 at 24 processes). Fix: each worker writes aready_<pid>file and waits for agofile that the driver writes once all workers are ready; throughput isruns / (last end - go time). Merged in FLU_USA PR 64.Also relevant:
parallelnodes than onsharednodes. Record the node and partition with every row.Suggested acceptance addition: the packed-start barrier and a per-row record of node, partition and
SLURM_CPUS_ON_NODEare part of the executor.jc-macdonald commentedon Oct 20, 2026
PR #213 implements the minimal frozen Python task launch contract: detached source, runtime/import checks, immutable plans, packed CPU tasks, guarded receipts and dependency handling. All eight final-head checks passed (902 local tests and strict docs). A small real Rockfish run passed: jobs 32273353 and 32273355 exercised two tasks and completed-receipt reuse; a failed dependency was refused. The first pilot used a login-only interpreter and failed before tasks; selecting the shared Python module fixed that site configuration.
Keeping this issue open for the broader executor work: direct integration into existing
run_grid/adaptive entry points, distributed result/shard merge semantics and broader target-site qualification are not implemented by this slice. The current API runs explicit committed Python entry points and delegates within-task recovery to existing grid/adaptive journals. It does not serialize arbitrary live evaluators.