Show the invocation that actually runs: REPO and RAMPNET_ENV (#51, #135) - #160
Open
jonfroehlich wants to merge 1 commit into
Open
Show the invocation that actually runs: REPO and RAMPNET_ENV (#51, #135)#160jonfroehlich wants to merge 1 commit into
jonfroehlich wants to merge 1 commit into
Conversation
Campaign B has now failed to train a single step three times, on three different one-line causes, and the launcher's own USAGE header steered into two of them by showing the bare `SEED=1 sbatch ...` form: 39515025/26/27 2026-09-03 bash parse error in the SEED guard (2b7471c) 39583887/90/91 2026-09-04 RAMPNET_ENV unset -> `source activate`, which a non-login sbatch environment cannot resolve 39619547/48/49 2026-09-06 REPO unset -> $HOME/RampNet, a stale Aug-17 clone whose train.py predates --seed, so all 16 ranks died with "unrecognized arguments: --seed 1" Each cost one to three days of calendar, because a bad submission returns a job id and is indistinguishable from a healthy one in squeue. The header and the doc's reproduce block now give the full form, and say why each variable is load-bearing: - REPO also feeds PYTHONPATH, so a wrong REPO hides rampnet.seeding too. The 30 KB RendezvousConnectionError traceback that fills the .err is downstream noise from the ranks tearing down; the argparse error is on line 1. Say so, because the tail is what you reach for first. - The env path takes a literal `jonf`, not $USER. /gscratch/makelab carries both `jonf` and `jfroehli` directories for the same person and the env is under `jonf`, while $USER is `jfroehli` -- correct for RUNDIR and DATA_ROOT on /gscratch/scrubbed, wrong here. - Verify with `sacct -D` about an hour after submit. Plain `sacct` hides requeued incarnations. Comments and docs only; no behaviour change. tests/test_seeding.py still passes (23 tests) and `bash -n` is clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nz62vjjbMSdmndABSj3JWq
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Campaign B of the seed-variance campaign (#135, and the number #51 is blocked on) has now failed to train a single step three times, on three different one-line causes. It was dead again from 2026-09-06 until this morning, and nothing was queued in between.
SEEDguard — fixed in 2b7471cRAMPNET_ENVunset →source activate sidewalkcv2, which a non-login sbatch environment cannot resolveREPOunset →$HOME/RampNet, a stale Aug-17 clone whosetrain.pypredates--seed, so all 16 ranks died withtrain.py: error: unrecognized arguments: --seed 1The launcher's own USAGE header steered into two of those, because it showed the bare
SEED=1 sbatch stage_two/run_train_seed.slurmform — which cannot work on klone as things stand.docs/seed_variance_51_135.md's reproduce block showed the same form, pluscd ~/RampNet, which is precisely the stale clone. This was flagged on #135 at the time and not fixed.Both now give the full invocation and say why each variable is load-bearing:
REPOalso feedsPYTHONPATH, so a wrongREPOhidesrampnet.seedingas well as the wrongtrain.py.$PWDis the right value whenever you submit from the checkout you mean to run..err; the remaining ~30 KB is a torchRendezvousConnectionErrortraceback from the ranks tearing down. The header says to read the head, not the tail, because the tail is what you reach for first and it names the wrong thing.jonf, not$USER./gscratch/makelabcarries both ajonfand ajfroehlidirectory for the same person, and the env is underjonfwhile$USERisjfroehli.$USERis correct forRUNDIRandDATA_ROOTon/gscratch/scrubbed, which is what makes this easy to miss.sacct -Dabout an hour after submit. All three failures returned a job id and were indistinguishable from a healthy submission insqueue; two sat unnoticed for a day or more. Plainsacctadditionally hides requeued incarnations.Comments and documentation only — no behaviour change.
tests/test_seeding.pypasses (23 tests) andbash -n stage_two/run_train_seed.slurmis clean.Campaign B was resubmitted with the corrected form this morning as 39880702 / 39880703 / 39880706.
🤖 Generated with Claude Code — Opus 5 (1M context), claude-opus-5[1m]