Add a guide for multi-node serving on Dynamo

docs/content/guides/serving-multi-node-on-dynamo.md

@@ -0,0 +1,120 @@
+---
+title: Multi-node serving on Dynamo
+weight: 30
+description: Serve a model too large for one GPU across two nodes, gang-scheduled by the Dynamo stack.
+---
+<!-- vale write-good.Passive = NO -->
+Qwen2.5-14B's FP16 weights are about 29 GB, larger than one NVIDIA L4's 23 GB, so
+it serves across two nodes as a gang: a `Leader` and a `Worker`, one L4 each,
+pipeline-parallel across the pair. On a
+[Dynamo cluster]({{< ref "/platform/inference-cluster.md#serving-stack" >}}) Grove
+and the KAI Scheduler gang-schedule the two pods together, Modelplane composes
+them as a Grove `PodCliqueSet`, and NVIDIA ModelExpress serves their weights.
+
+This is the [getting started tour]({{< ref "/getting-started" >}}) scaled to two
+nodes: one larger model on a `spec.stack: Dynamo` cluster.
+[Set up the platform]({{< ref "/getting-started/build-the-platform.md" >}}) first,
+for the gateway and cloud credentials, then apply the manifests below.
+
+## Register a Dynamo cluster
+
+The `InferenceClass` describes a single-L4 node, sized up from the getting started
+tour's for the larger model. The `InferenceCluster` runs two of them and sets
+`spec.stack: Dynamo`, so Modelplane installs Grove, the KAI Scheduler, and
+ModelExpress on the cluster.
+
+{{< manifests "guides/serving-multi-node-on-dynamo/inference-class.yaml" >}}
+
+{{< manifests "guides/serving-multi-node-on-dynamo/inference-cluster.yaml" >}}
+
+Provisioning the pool and installing the stack takes about 15 minutes:
+
+```bash
+kubectl wait --for=condition=Ready ic/eks-us-east --timeout=20m
+```
+
+## Cache the weights
+
+A gang reads its weights from a shared cache, so pods don't each pull a copy.
+Create the namespace and the cache:
+
+```bash
+kubectl create namespace ml-team
+```
+
+{{< manifests "guides/serving-multi-node-on-dynamo/model-cache.yaml" >}}
+
+## Deploy the gang
+
+The `Leader` and `Worker` run the same `vllm serve`, differing only in node rank.
+`$(MODELPLANE_LEADER_ADDRESS)` resolves to the leader on either stack, but
+`$(MODELPLANE_RANK)` isn't injected on Dynamo yet, so the worker derives its rank
+from Grove's `GROVE_PCLQ_POD_INDEX`.
+[Multi-node deployments]({{< ref "/models/model-deployment.md#multi-node" >}})
+covers this. Both load with `--load-format modelexpress`, so they read the cached
+weights through the Dynamo stack's ModelExpress server.
+
+{{< manifests "guides/serving-multi-node-on-dynamo/model-deployment.yaml" >}}
+
+Wait until `READY` shows `True`. The first start hydrates the cache, so it's
+slower than later ones:
+
+```bash
+kubectl get md -n ml-team --watch
+```
+
+On the workload cluster the gang is a Grove `PodCliqueSet`, the Dynamo stack's
+multi-node workload in place of a LeaderWorkerSet:
+
+```bash
+kubectl get podcliquesets.grove.io -A   # workload cluster
+```
+
+## Expose and query
+
+{{< manifests "guides/serving-multi-node-on-dynamo/model-service.yaml" >}}
+
+Read the endpoint's address and send it a request. The `model` field is the
+`--served-model-name` the deployment sets:
+
+```bash
+ADDRESS=$(kubectl get ms qwen2-5-14b -n ml-team -o jsonpath='{.status.address}')
+kubectl run -i --rm curl-test \
+  --image=curlimages/curl \
+  --restart=Never \
+  --env="ADDRESS=$ADDRESS" \
+  -- sh -c 'curl -s "$ADDRESS/v1/chat/completions" \
+  -H "Content-Type: application/json" \
+  -d "{\"model\":\"qwen2.5-14b\",\"messages\":[{\"role\":\"user\",\"content\":\"What is Kubernetes in one sentence?\"}],\"max_tokens\":100}"'
+```
+
+The request routes through the gateway to the leader, which serves the gang's one
+endpoint.
+
+## Scale out with peer-to-peer loading
+
+Add a second replica and ModelExpress shows what it's for. The first replica
+seeds its weights from the cache and publishes itself as a source; the second
+loads them straight from the first, peer-to-peer, rather than reading the cache
+again.
+
+Each replica is a gang of two nodes, so a second replica needs two more nodes.
+Grow the pool to four, then scale the deployment:
+
+```bash
+kubectl patch ic/eks-us-east --type=json -p '[
+  {"op":"replace","path":"/spec/nodePools/0/nodeCount","value":4},
+  {"op":"replace","path":"/spec/nodePools/0/minNodeCount","value":4},
+  {"op":"replace","path":"/spec/nodePools/0/maxNodeCount","value":4}]'
+kubectl patch md/qwen2-5-14b -n ml-team --type=merge -p '{"spec":{"replicas":2}}'
+```
+
+Once the second gang starts, watch its leader load from the first over
+ModelExpress:
+
+```bash
+kubectl logs -n default -l modelplane.ai/clique-role=leader -c engine --tail=-1 \
+  | grep "source worker"   # workload cluster
+# [Worker 0] Trying source worker 63d91022 (266 tensors)
+```
+<!-- vale write-good.Passive = YES -->

docs/manifests/guides/serving-multi-node-on-dynamo/inference-class.yaml

@@ -0,0 +1,27 @@
+# EKS g6.8xlarge, one NVIDIA L4 per node. A single L4 can't hold the 14B model,
+# so the gang spans two nodes. This node has enough memory to hold the weights as
+# they load, and the 100 GB disk holds the vLLM image.
+apiVersion: modelplane.ai/v1alpha1
+kind: InferenceClass
+metadata:
+  name: l4-1x-g6
+spec:
+  description: "EKS g6.8xlarge, 1x NVIDIA L4"
+  provisioning:
+    provider: EKS
+    eks:
+      instanceType: g6.8xlarge
+      diskSizeGb: 100
+      accelerator:
+        type: nvidia-l4
+        count: 1
+  devices:
+  - name: gpu
+    claim: DRA
+    driver: gpu.nvidia.com
+    deviceClassName: gpu.nvidia.com
+    count: 1
+    attributes:
+      architecture: { string: Ada Lovelace }
+    capacity:
+      memory: { value: "23034Mi" }   # L4's real reported VRAM (not the nominal 24GB)

docs/manifests/guides/serving-multi-node-on-dynamo/inference-cluster.yaml

@@ -0,0 +1,25 @@
+# An EKS cluster running the Dynamo serving stack, with a two-node L4 pool so a
+# gang can span both nodes. spec.stack: Dynamo installs Grove and the KAI
+# Scheduler, which gang-schedule the leader and worker together and compose them
+# as a Grove PodCliqueSet, and ModelExpress, which serves cached weights to the
+# gang and moves them between replicas peer-to-peer.
+apiVersion: modelplane.ai/v1alpha1
+kind: InferenceCluster
+metadata:
+  name: eks-us-east
+  labels:
+    modelplane.ai/region: us-east
+spec:
+  stack: Dynamo
+  cluster:
+    source: EKS
+    eks:
+      region: us-east-1
+  nodePools:
+  - name: gpu-l4
+    className: l4-1x-g6
+    nodeCount: 2
+    minNodeCount: 2
+    maxNodeCount: 2
+    zones:
+    - us-east-1b

docs/manifests/guides/serving-multi-node-on-dynamo/model-cache.yaml

@@ -0,0 +1,14 @@
+# The shared read-write-many cache the gang serves from, hydrated once from
+# Hugging Face. Both gang pods mount it and read weights from it over EFS,
+# instead of each pulling its own copy. Qwen2.5-14B is open, so it needs no
+# token. Its FP16 weights are about 29 GB, so sizeGiB leaves headroom.
+apiVersion: modelplane.ai/v1alpha1
+kind: ModelCache
+metadata:
+  name: qwen2-5-14b
+  namespace: ml-team
+spec:
+  source: HuggingFace
+  huggingFace:
+    repo: Qwen/Qwen2.5-14B-Instruct
+    sizeGiB: 40

docs/manifests/guides/serving-multi-node-on-dynamo/model-deployment.yaml

@@ -0,0 +1,101 @@
+# Qwen2.5-14B served across two L4 nodes as a gang. The FP16 weights (~29 GB)
+# don't fit one L4's 23 GB, so the engine is a Leader + Worker gang,
+# pipeline-parallel across two g6.8xlarge nodes with one L4 each. Both pods mount
+# the shared ModelCache.
+#
+# The cluster runs the Dynamo stack, so Grove and the KAI Scheduler gang-schedule
+# the two pods, and Modelplane composes them as a Grove PodCliqueSet.
+# $(MODELPLANE_LEADER_ADDRESS) resolves to the leader on Dynamo, but
+# $(MODELPLANE_RANK) isn't injected there yet (modelplaneai/modelplane#418), so
+# each command sets its own --node-rank: 0 on the leader, and
+# $$((GROVE_PCLQ_POD_INDEX + 1)) on the worker ($$ escapes past Kubernetes,
+# leaving $((...)) for the shell to evaluate).
+#
+# Notes on the engine flags:
+#   --pipeline-parallel-size=2 splits the model across the two nodes;
+#     --tensor-parallel-size=1 keeps one GPU per node. Pipeline parallelism sends
+#     only activations between nodes, so it stays light on the network.
+#   --distributed-executor-backend=mp is vLLM's native multiprocessing multi-node
+#     path; vllm/vllm-openai:v0.23.0 no longer ships Ray.
+#   --load-format modelexpress loads weights through the ModelExpress server the
+#     Dynamo stack runs. The first replica seeds from the cache and publishes
+#     itself; later replicas pull their weights peer-to-peer from a replica that
+#     already has them. The vLLM image doesn't ship the loader, so pip install it
+#     first. --load-format=runai_streamer is the alternative that always reads the
+#     cache directly, on any stack.
+#   --max-model-len=8192 caps context so the KV cache fits alongside the weights.
+apiVersion: modelplane.ai/v1alpha1
+kind: ModelDeployment
+metadata:
+  name: qwen2-5-14b
+  namespace: ml-team
+spec:
+  replicas: 1
+  template:
+    spec:
+      modelCacheRef:
+        name: qwen2-5-14b
+      engines:
+      - name: qwen
+        members:
+        - role: Leader
+          nodeSelector:
+            devices:
+            - name: gpu
+              count: 1
+              selectors:
+              - cel: |
+                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("20Gi")) >= 0
+          template:
+            spec:
+              containers:
+              - name: engine
+                image: vllm/vllm-openai:v0.23.0
+                command:
+                - /bin/sh
+                - -c
+                - >-
+                  pip install --index-url https://pypi.nvidia.com modelexpress &&
+                  exec vllm serve Qwen/Qwen2.5-14B-Instruct
+                  --served-model-name=qwen2.5-14b
+                  --tensor-parallel-size=1
+                  --pipeline-parallel-size=2
+                  --distributed-executor-backend=mp
+                  --nnodes=2 --node-rank=0
+                  --master-addr=$(MODELPLANE_LEADER_ADDRESS)
+                  --load-format modelexpress
+                  --max-model-len=8192
+                  --gpu-memory-utilization=0.90
+                  --port=8000
+        - role: Worker
+          worker:
+            nodes: 1
+          nodeSelector:
+            devices:
+            - name: gpu
+              count: 1
+              selectors:
+              - cel: |
+                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("20Gi")) >= 0
+          template:
+            spec:
+              containers:
+              - name: engine
+                image: vllm/vllm-openai:v0.23.0
+                command:
+                - /bin/sh
+                - -c
+                - >-
+                  pip install --index-url https://pypi.nvidia.com modelexpress &&
+                  exec vllm serve Qwen/Qwen2.5-14B-Instruct
+                  --served-model-name=qwen2.5-14b
+                  --tensor-parallel-size=1
+                  --pipeline-parallel-size=2
+                  --distributed-executor-backend=mp
+                  --nnodes=2 --node-rank=$$((GROVE_PCLQ_POD_INDEX + 1))
+                  --master-addr=$(MODELPLANE_LEADER_ADDRESS)
+                  --headless
+                  --load-format modelexpress
+                  --max-model-len=8192
+                  --gpu-memory-utilization=0.90
+                  --port=8000

docs/manifests/guides/serving-multi-node-on-dynamo/model-service.yaml

@@ -0,0 +1,14 @@
+# Exposes the gang as one OpenAI-compatible URL. Modelplane composes one
+# ModelEndpoint per replica, labeled modelplane.ai/deployment: qwen2-5-14b, so
+# this selector reaches it. Read the public address from status.address:
+#   kubectl get ms qwen2-5-14b -n ml-team -o jsonpath='{.status.address}'
+apiVersion: modelplane.ai/v1alpha1
+kind: ModelService
+metadata:
+  name: qwen2-5-14b
+  namespace: ml-team
+spec:
+  endpoints:
+  - selector:
+      matchLabels:
+        modelplane.ai/deployment: qwen2-5-14b

docs/utils/vale/styles/config/vocabularies/Modelplane/accept.txt

@@ -251,6 +251,7 @@ H100
 H200
 EFA
 BF16
+FP16
 FP8
 INT4
 AWQ
@@ -530,6 +531,8 @@ all-or-nothing
 [Pp]eer-to-peer
 [Gg]ang-schedules?
 routable
+pipeline-parallel
+Qwen2.5-14B
 
 agent
 requirement