[Backport release-0.4] Add a guide for multi-node serving on Dynamo - #451
Merged
Merged
Conversation
The getting started tour deploys a single-node model, and nothing walks through a multi-node gang on the Dynamo stack: how the leader and worker land as a Grove PodCliqueSet, how a gang command adapts where MODELPLANE_RANK isn't injected on Dynamo yet, and how NVIDIA ModelExpress moves weights between replicas. This adds a guide modeled on the getting started tour, scaled to two nodes. Qwen2.5-14B's FP16 weights don't fit one L4, so it serves across two g6.8xlarge nodes as a Leader + Worker gang, pipeline-parallel. The cluster sets spec.stack: Dynamo, so Grove and the KAI Scheduler gang-schedule the pods and ModelExpress serves their weights. A closing step scales to a second replica that loads peer-to-peer from the first. Validated end to end on EKS: the gang serves Qwen2.5-14B, and a second replica loads its weights peer-to-peer over ModelExpress. Towards #111. Signed-off-by: Nic Cope <nicc@rk0n.org> (cherry picked from commit c5fdd45)
3 tasks
haarchri
approved these changes
Sep 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Backport of #450 to
release-0.4.I have:
nix flake check(or./nix.sh flake check) and made sure it passes.git commit -s.