[scttfrdmn]

Fixes #19

Refs spore-host/spore-host#644 (pre-1.0 hardening).

Root cause

_task_id built smk-<job.name>-<attempt>, and job.name is the rule name. Every job of a wildcard fan-out therefore got the same id (e.g. four stats jobs → smk-stats-1). The id is both the instance name and the key of tasks/<id>/completion.json. check_active_jobs polls that one key for every SubmittedJobInfo, so with --jobs N > 1 each job was reported from whichever shard's record was there at that moment. A failed shard could be reported as succeeded, or the reverse. Each new launch also clears the shared record (tasklaunch.clearStaleCompletion), which explains the delayed, batched completions measured in the issue comment.

Fix

The id is now smk-<rule>-<jobid>-<attempt>. This is the issue's suggested primitive: jobid is already in the dispatch log line.

• Unique per job: jobid is unique per DAG node within a run (e.g. 2–5 for four stats shards).
• Stable across retries: Snakemake retries the same job object, so jobid is unchanged and only attempt moves. A retry gets a fresh record and never reads the previous attempt's, as before.
• Truncation can't collide: the 60-char cap now applies only to the rule part. The -<jobid>-<attempt> suffix is always kept, so a long rule name can't truncate two jobs into one id again. Group jobs (UUID jobids) are sanitized to the same [a-z0-9-] alphabet.
• Why not wildcards: they are friendlier to read, but long, unbounded, and they reintroduce the truncation-collision risk. jobid is short and sufficient.

Compatibility with existing records

• Records written under old ids (smk-<rule>-<attempt>) are never read by the new code. Snakemake doesn't persist external job ids across a restart of the main process, so no in-flight job can straddle the upgrade.
• Old records remain in spawn-results-<acct>-<region>/tasks/smk-*. Nothing reads them and they can be deleted.
• An old-form id can textually equal a new one, e.g. rule stats_4 attempt 1 (old smk-stats-4-1) and rule stats, jobid 4, attempt 1 (new). This is harmless: spawn task run deletes any stale completion record for the id before launching (clearStaleCompletion), so a new job never reads an old record.
• jobid restarts at 0 in each run, so two separate runs of the same workflow reuse ids, as they did before with rule names. That is safe for the same reason (launch clears the stale record). It only matters for concurrent runs of the same workflow under the same account and region, which were already unsafe and are no worse now.

Tests

tests/test_task_id_per_job.py (6 tests) all fail on main and pass with the fix. The key one drives run_job for four shards of one rule and then check_active_jobs with one shard's record failing, and asserts that exactly that shard errors. Full suite: 50 passed. ruff check and mypy are clean.

The engine-composition test's seeded id changes to smk-noout-0-1: the only job of a one-job DAG is jobid 0, confirmed with snakemake -n. The Composition CI job on this PR exercises it against Substrate.

Before merge

A live fan-out (the cookbook Shape-F recipe, --jobs 4) is worth running. Expect one tasks/smk-stats-<jobid>-1/ prefix per shard, and completions reported per job rather than in batches.

[scttfrdmn]

Live verification: PASS (spore-host-dev, 2026-11-08)

Setup.

• Plugin installed from this PR's head 2888f39 into a venv with snakemake 9.27.1 and snakemake-storage-plugin-s3. spawn v0.161.0, account 435415984226, us-east-1.
• Command: --executor spawn --jobs 4 --keep-going --spawn-instance-type t4g.small --spawn-ttl 1h --spawn-cost-limit 0.10, with S3 default storage.
• stats is a 4-way wildcard fan-out (shards a/b/c/d) that sleeps 0/40/80/120 s so the shards finish at different times. Shard c fails on purpose, which makes a shared record observable.

One id and one record per shard:

spawn: dispatching smk-stats-5-1 via `spawn task run` for job 5   (d)
spawn: dispatching smk-stats-4-1 via `spawn task run` for job 4   (c)
spawn: dispatching smk-stats-3-1 via `spawn task run` for job 3   (b)
spawn: dispatching smk-stats-2-1 via `spawn task run` for job 2   (a)

$ aws s3 ls s3://spawn-results-435415984226-us-east-1/tasks/ | grep smk-stats
PRE smk-stats-2-1/  PRE smk-stats-3-1/  PRE smk-stats-4-1/  PRE smk-stats-5-1/

smk-stats-2-1  exit 0 completed  21:39:11Z -> 21:40:24Z
smk-stats-3-1  exit 0 completed  21:38:58Z -> 21:40:49Z
smk-stats-4-1  exit 1 failed     21:38:48Z -> 21:41:18Z
smk-stats-5-1  exit 0 completed  21:38:37Z -> 21:41:46Z

Completions are reported per job, one at a time, a few seconds after each record:

13:40:28 Finished jobid: 2 (Rule: stats)                 record ended 21:40:24Z
13:40:58 Finished jobid: 3 (Rule: stats)                 record ended 21:40:49Z
13:41:22 Error in rule stats: ... jobid: 4 ... external_jobid: smk-stats-4-1
13:41:56 Finished jobid: 5 (Rule: stats)                 record ended 21:41:46Z

• Exactly the failing shard (c, jobid 4) errored, and the three good shards succeeded. Under the old shared smk-stats-1 id, a failure here would have been reported against whichever shard polled that record.
• S3 holds stats/a.txt, stats/b.txt and stats/d.txt, and no c.txt. summary correctly did not run (--keep-going, so it exits 1).
• One minor observation: the record's exit code is 1, while the shell's was exit 3. Snakemake's on-instance job wrapper collapses it. This is not related to this PR.

Cost: 4 × t4g.small for about 1.5–3 min each, all self-terminated.

Leak check (us-east-1 and us-west-2, after teardown).

• No instances pending, running, stopping, stopped or shutting-down. No open or active Spot requests.
• No unattached volumes; the test snapshot snap-0120c9d0f4e4384fd and its source volume are deleted.
• No unassociated EIPs, no new security groups or key pairs, and no spawn-pool* SQS queues.
• The scratch bucket nf-live-435415984226-20261108 is removed, as are this run's tasks/* and staging/pool-specs/* records in spawn-results-435415984226-us-east-1.
• The per-task role spawn-instance-1342ea0f remains (spawn-managed, removed by spawn cleanup --instance-roles or the reaper after the 24 h guard). The pool roles were already gone.