Bugfix - Use torchrun --standalone for single-node torch.distributed runs - #839
Bugfix - Use torchrun --standalone for single-node torch.distributed runs#839polarG wants to merge 2 commits into
Conversation
Single-node runs passed no rendezvous args, so torchrun used the fixed default port 29500, causing transient EADDRINUSE collisions between consecutive distributed benchmarks (e.g. model-benchmarks:resnet). Use --standalone so torchrun binds a random free port. Multi-node path unchanged; runner unit test expectation updated.
There was a problem hiding this comment.
🟢 Ready to approve
The change is small, targeted to the reported failure mode, preserves the multi-node behavior, and the affected unit test expectation is updated accordingly.
This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.
Pull request overview
This PR fixes intermittent single-node torchrun failures in the runner’s torch.distributed mode by ensuring single-node runs use torchrun --standalone (random free rendezvous port) rather than relying on torchrun’s default rendezvous behavior (fixed MASTER_PORT=29500), which can race with TIME_WAIT and cause EADDRINUSE.
Changes:
- Update
SuperBenchRunner.__get_mode_commandto emittorchrun --standalonewhennode_num == 1fortorch.distributedmode. - Keep the existing multi-node rendezvous argument path unchanged.
- Update runner unit test expectations to include
--standalonefor the single-node case.
File summaries
| File | Description |
|---|---|
| superbench/runner/runner.py | Adds --standalone to single-node torch.distributed torchrun command generation to avoid fixed-port reuse failures. |
| tests/runner/test_runner.py | Updates test_get_mode_command expected command string for single-node torch.distributed runs to include --standalone. |
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 0
- Review effort level: Low
We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #839 +/- ##
=======================================
Coverage 86.02% 86.02%
=======================================
Files 103 103
Lines 7950 7950
=======================================
Hits 6839 6839
Misses 1111 1111
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
torch pulls in an unpinned setuptools during 'pip install .[test,cpuworker]', which upgrades the 65.7 pinned by the pipeline to 83.0.0 and makes the setup.py guard raise VersionConflict on the python-3.10 lint job. setuptools 66+ only drops Python 3.7, so keep the cap there only.
There was a problem hiding this comment.
🟢 Ready to approve
The changes are small, targeted, and the updated behavior is covered by an existing unit test that was updated to match the new command.
This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.
Review details
- Files reviewed: 3/3 changed files
- Comments generated: 0 new
- Review effort level: Lite
We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.
Description
When the runner builds a
torchruncommand for atorch.distributed-mode benchmark withnode_num == 1, it passes no rendezvous arguments at all. In that casetorchrunfalls back to its own default rendezvous, which binds a fixed port (MASTER_PORT=29500). If a previous distributed job's socket on that same port hasn't fully cleared the kernel'sTIME_WAITstate yet, the next single-node job'storchrunfails to bind with:This was observed for
model-benchmarks:resnetduring a fullsb runon an MI300X host (Ansible-level failure, before the benchmark executor even started, so no result appears in the results summary at all). This PR switches the single-node case totorchrun --standalone, which spins up its own local rendezvous on a randomly chosen free port instead of a fixed one, eliminating the port-reuse race entirely.Why this wasn't caught earlier
This is not a new bug: the
node_num == 1→ empty rendezvous-args code path has existed since thetorch.distributedmode was first added to the runner in 2021 (#81). It never surfaced as a visible failure before because it's a timing-dependent race, not a deterministic defect in any specific benchmark:superbench/config/amd_mi300.yamlshowsgpt,bert,lstm,resnet,densenet, andvggundermodel-benchmarksall share the identicaldefault_pytorch_modeanchor (torch.distributed,proc_num: 8,node_num: 1), so every one of them goes through the exact same fixed-port code path — yet onlyresnethitEADDRINUSEin the observed run, whilegpt/bert/lstmpassed. That rules out anything resnet-specific.TIME_WAIT) before the next single-node job attempts to bind. That gap is a function of exact benchmark scheduling/ordering and per-run timing, so it only manifests when two single-node jobs happen to land close enough together — which is why this only showed up on this particular full end-to-end 462-benchmark run and not in smaller/prior runs.Major Revision
superbench/runner/runner.py(__get_mode_command): fortorch.distributedmode withnode_num == 1, pass--standaloneinstead of no rendezvous args. Multi-node path (--nnodes=$NNODES --node_rank=$NODE_RANK --master_addr=$MASTER_ADDR --master_port=$MASTER_PORT) is unchanged.Minor Revision
tests/runner/test_runner.py: updatedtest_get_mode_command's expected single-node command string to include--standalone.Testing
test_get_mode_commandpasses with the updated expectation.torchrun --nproc_per_node=2(old default) always bindsMASTER_PORT=29500; with--standalone, back-to-back single-node runs picked distinct random ports (45635, then 36937) with noEADDRINUSE.