Stabilize PP8+DSpark token transport and draft execution - #1
Draft
nilesjarvis wants to merge 5 commits into
Draft
Conversation
…idation Replace the data-dependent sampled-token broadcast protocol in PPHandler with a fixed four-broadcast-per-step schedule (header + sampled + counts + draft), a per-step sequence header validated on every receiver, and sentinel payloads for mask-None or skipped-proposal steps. The upstream protocol gated the send on the last rank's own mask while receivers posted the draft recv unconditionally, which FIFO-mismatched collectives and silently hung the pipeline. Receivers now raise RuntimeError with the expected vs actual header the moment a step is skipped cross-rank, instead of deadlocking the engine. Also import faulthandler in WorkerProc so SIGUSR1 dumps all thread stacks to the journal without ptrace (diagnostics parity).
Wire header fixes on top of the validated PP broadcast protocol: - int64 header instead of int32: mask_hash is an unsigned CRC32 (0..2^32-1); int32 torch.tensor raised RuntimeError on ~50% of distinct masks (values >= 2^31). 64-bit seq also removes the 32-bit split. - Include num_reqs in the header: payload shapes are locally determined per rank ([num_reqs, ...]) and crc32(packbits(mask)) collides across batch sizes (e.g. all-False masks), so a num_reqs mismatch could pass validation and post mismatched-shape payloads, hanging the NCCL FIFO -- the exact wedge class this protocol exists to fail fast on. - Sender increments step_seq before enqueueing the header so sender and receivers advance at the identical protocol point. - record_stream the recv-side header past the blocking .cpu() read. Adds tests/v1/worker/gpu/test_pp_utils_header.py (unittest, 11 tests: dtype/width, high-bit CRC32 roundtrip, seq placement, num_reqs disambiguation, validation equality semantics).
…park With async scheduling + V2 runner, max_concurrent_batches = pp_size+1. The extra in-flight batch lets a sample_tokens(N+1) RPC race ahead of the last PP rank's sample_tokens(N): flight-recorder dumps show non-last ranks one draft-broadcast sequence ahead (enq N+1) while the last rank is still at N, deadlocking the pp_broadcast group (60s NCCL watchdog, EngineDead). Guarded opt-in (VLLM_WEDGE_AB_CAP_CONCURRENT=1, pp_size>1, method=dspark): cap at pp_size, keeping async_scheduling=True and the V2 PP token path. Default behavior unchanged for every other configuration. Validated: previously-reliable ~60s wedge trigger (max_tokens overflow at temp 1.0) passed 3x consecutively, then a 66-minute sustained stress pass (200K-token prefills, 2-4 concurrent tool-schema streams, 25 overflow rounds) with zero wedges/watchdog events; baseline boots wedged within minutes on every attempt.
- test_pp_receive_posts_sample_metadata_and_real_draft_broadcasts: fill the int64 validation header the way the sender does; assert 4 broadcasts [(3,),(1,6),(2,1),(1,5)], step_seq advance, and the fail-fast desync raise on a stale header. - tests/config/test_max_concurrent_batches_cap.py: 8 cases proving the in-flight cap applies only with VLLM_WEDGE_AB_CAP_CONCURRENT=1 + pp_size>1 + method=dspark, and that PP=1 async (2), non-dspark (9), env-off (9), V1-runner and non-async values are unchanged.
Author
|
Reproducer of residual failure (2026-08-23 06:31 UTC, capped config |
nilesjarvis
marked this pull request as draft
August 23, 2026 06:44
Add an opt-in dedicated Gloo transport for the tiny sampled-token protocol, defer draft receives until consumption, and allow DSpark draft-only eager execution while retaining target CUDA graphs.
nilesjarvis
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Recurring production deadlock on 8x RTX 3090 (SM86), PP8 + DSpark
(
method=dspark, 5 spec tokens), V2 runner, and async scheduling. One requestfreezes, PP ranks stop completing
sample_tokens, and EngineCore eventuallytimes out. The failure reproduced repeatedly under real LAN traffic and the
included stress harness.
Flight-recorder dumps, distributed watchdog output, and live SIGUSR1 worker
stacks exposed multiple independent hazards:
PPHandlerhad a data-dependent collective schedule with no rendezvousvalidation.
pp_size + 1in-flight window allowed non-last ranks to enterthe next token-propagation step before PP7.
communicator could stall on the same final-draft sequence while activation
P2P was active.
final draft while PP7 waited for GPU work before sending it.
stall to PP7's DSpark full CUDA-graph proposal. The sampled target output had
already become host-visible; PP0-6 had returned to their RPC loops.
Changes
8ce1378): every step posts exactly fourpayloads (header, sampled tokens, counts, drafts), using sentinels when no
real output exists.
fa93533): int64 sequence/hash header,num_reqsvalidation, aligned sender/receiver sequence advancement, and fail-fast
desync errors.
1900969): cap V2 PP+DSpark async batches atpp_sizewithVLLM_WEDGE_AB_CAP_CONCURRENT=1.6ea9ae6):CPU buffer/work handle until consumption;
pending batches;
VLLM_WEDGE_AB_DSPARK_EAGER=1, while the target model retainsFULL_DECODE_ONLYCUDA graphs;isolation.
The dedicated CPU path is enabled with
VLLM_WEDGE_AB_PP_BROADCAST_CPU=1. All three fallbacks are opt-in; defaultbehavior for other configurations is unchanged.
Validation
Final live configuration: PP8, DSpark N=5, async scheduling, target
FULL_DECODE_ONLY, cap + dedicated/deferred Gloo + DSpark-only eager.70,986 generated tokens, zero stalls, zero HTTP 500, zero restarts, queues
drained.
3,900-second deadline), 24 sustained rounds, 14 fresh salted >=200K-token
prefills, concurrent streaming/tool-schema requests, 25
finish=lengthoverflow cases, and a short-stop case. Result:
abort=null, 93,746 generatedtokens, 35,924 engine iterations, final running=0/waiting=0, zero watchdog or
fatal journal matches, zero HTTP 500, zero restarts, and no kernel Xid event.
async_op=Truereturned immediately while the sender was delayed, and the retained work
later produced the correct payload.
sibling Gloo membership was verified independently. Ruff, format, diff, and
compile checks pass for all nine changed files.
each other; 8/10 outputs match the original
ccfeb3abaseline byte-for-byte.The two differences are coherent alternate wording/truncation, not corrupt
output.
The earlier claim that the cap alone passed a 66-minute run was falsified by a
later exact-shape run at ~2,627 seconds; this PR description supersedes that
result. The final fallback passed the same round-14/tenth-long-prefill position
and the full validation above.
Known separate issue
--no-async-schedulingis not a usable workaround in this fork: the firstreal PP+DSpark sample can hit a separate
logits_indices=-1CUDA assertion.That path remains outside this change.