Add a guide for multi-node serving on Dynamo

[negz]

Description

Adds a hands-on guide for serving a model too large for one GPU across two nodes as a gang on the Dynamo serving stack, modeled on the getting started tour scaled to two nodes. Qwen2.5-14B's FP16 weights don't fit one L4, so it serves as a Leader + Worker gang, pipeline-parallel, on a spec.stack: Dynamo cluster where Grove and the KAI Scheduler gang-schedule the pods as a Grove PodCliqueSet and ModelExpress serves their weights.

The guide teaches the two things that are specific to a multi-node gang on Dynamo:

• The gang-command difference: MODELPLANE_RANK isn't injected on Dynamo yet, so the worker derives its rank from Grove's GROVE_PCLQ_POD_INDEX.
• NVIDIA ModelExpress (--load-format modelexpress): a closing step scales to a second replica that loads its weights peer-to-peer from the first, rather than reading the cache again.

Validation

Validated end to end on EKS (g6.8xlarge, 1x L4 per node):

• The two-node gang forms as a Grove PodCliqueSet, gang-scheduled across both nodes; the GROVE_PCLQ_POD_INDEX rank derivation and MODELPLANE_LEADER_ADDRESS resolve correctly.
• The deployment serves Qwen2.5-14B end to end through the gateway (system_fingerprint: vllm-...-pp2).
• Scaling to a second replica, the new gang loads its weights peer-to-peer from the first over ModelExpress (Trying source worker ... Transfer complete: 266 tensors, 14.78 GB), not from the cache.

Found and fixed along the way: the model name can't contain a dot (Grove derives a headless Service from it), and the nodes need more disk and memory than the tour's default to hold the vLLM image and load a 14B pipeline shard.

Towards #111.

I have:

• Run nix flake check to test my change.
• Added or updated tests where necessary. (docs-only; manifests pass schema validation)
• Ensured my code is readable and well-commented.

[Copilot]

🟡 Changes recommended

Unresolved critical naming and moderate cluster-scoping issues block approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a guide for serving Qwen2.5-14B across two Dynamo-managed L4 nodes.

Changes:

• Adds Dynamo/EKS, cache, deployment, and service manifests.
• Documents Grove gang scheduling and ModelExpress loading.
• Updates Vale vocabulary for new terminology.

File summaries

File	Summary
docs/utils/vale/styles/config/vocabularies/Modelplane/accept.txt	Adds accepted terminology.
docs/manifests/guides/serving-multi-node-on-dynamo/model-service.yaml	Defines the gateway-facing service.
docs/manifests/guides/serving-multi-node-on-dynamo/model-deployment.yaml	Defines the two-node vLLM gang; moderate: lacks unique cluster selection.
docs/manifests/guides/serving-multi-node-on-dynamo/model-cache.yaml	Defines model caching; moderate: lacks cluster scoping.
docs/manifests/guides/serving-multi-node-on-dynamo/inference-cluster.yaml	Defines the Dynamo cluster; critical: names collide with getting-started manifests.
docs/manifests/guides/serving-multi-node-on-dynamo/inference-class.yaml	Defines the L4 node class.
docs/content/guides/serving-multi-node-on-dynamo.md	Documents the end-to-end multi-node serving workflow.

Review details

Suppressed comments (1)

docs/manifests/guides/serving-multi-node-on-dynamo/model-cache.yaml:14

• With no clusterSelector, this cache stages 40 GiB on every InferenceCluster in the fleet. The guide explicitly starts from the getting-started platform, so applying it can hydrate this 29 GB model on unrelated Standard/L40S/A100 clusters, adding substantial storage and startup cost. Scope the cache to the same uniquely labeled Dynamo cluster used by the deployment.

spec:
  source: HuggingFace
  huggingFace:
    repo: Qwen/Qwen2.5-14B-Instruct
    sizeGiB: 40

• Files reviewed: 7/7 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[dennis-upbound]

Nice guide, and the manifests line up with the engine contracts. Approving and merging, since you're out of cell service.

One thing for the record rather than the branch: the description says the pool sets fabric: EFA, so the gang's cross-node traffic and ModelExpress's peer-to-peer transfer run over an RDMA fabric. The manifests don't set fabric at all, so the pool is on standard VPC TCP.

That looks deliberate and right. EFA is only useful on EFA-capable instance types and g6.8xlarge isn't one, and the title already dropped it. Flagging it so the description doesn't outlive the decision, and because the scale-out section otherwise reads as though the peer-to-peer load it shows is going over RDMA. Worth a quick edit to the body when you're back.

The rest checked out, including the parts that fail quietly if they're wrong: modelplane.ai/clique-role=leader is set on the leader clique and Grove propagates it to the pods, so the log command finds something; the CEL 20Gi floor passes against the class's declared 23034Mi; minNodeCount: 2 keeps the GPU pool off the DRA scale-from-zero path; the scale-out arithmetic lands correctly on four nodes for two two-node gangs; and --load-format modelexpress against a bare repo name works because cache_env sets HF_HUB_CACHE and modelexpress_env falls back to it.

Taken on trust: the vLLM 0.23 flag names and the run itself.

[github-actions]

Successfully created backport PR for release-0.4:

• [Backport release-0.4] Add a guide for multi-node serving on Dynamo #451