diff --git a/docs/content/guides/serving-multi-node-on-dynamo.md b/docs/content/guides/serving-multi-node-on-dynamo.md new file mode 100644 index 000000000..a5437c549 --- /dev/null +++ b/docs/content/guides/serving-multi-node-on-dynamo.md @@ -0,0 +1,120 @@ +--- +title: Multi-node serving on Dynamo +weight: 30 +description: Serve a model too large for one GPU across two nodes, gang-scheduled by the Dynamo stack. +--- + +Qwen2.5-14B's FP16 weights are about 29 GB, larger than one NVIDIA L4's 23 GB, so +it serves across two nodes as a gang: a `Leader` and a `Worker`, one L4 each, +pipeline-parallel across the pair. On a +[Dynamo cluster]({{< ref "/platform/inference-cluster.md#serving-stack" >}}) Grove +and the KAI Scheduler gang-schedule the two pods together, Modelplane composes +them as a Grove `PodCliqueSet`, and NVIDIA ModelExpress serves their weights. + +This is the [getting started tour]({{< ref "/getting-started" >}}) scaled to two +nodes: one larger model on a `spec.stack: Dynamo` cluster. +[Set up the platform]({{< ref "/getting-started/build-the-platform.md" >}}) first, +for the gateway and cloud credentials, then apply the manifests below. + +## Register a Dynamo cluster + +The `InferenceClass` describes a single-L4 node, sized up from the getting started +tour's for the larger model. The `InferenceCluster` runs two of them and sets +`spec.stack: Dynamo`, so Modelplane installs Grove, the KAI Scheduler, and +ModelExpress on the cluster. + +{{< manifests "guides/serving-multi-node-on-dynamo/inference-class.yaml" >}} + +{{< manifests "guides/serving-multi-node-on-dynamo/inference-cluster.yaml" >}} + +Provisioning the pool and installing the stack takes about 15 minutes: + +```bash +kubectl wait --for=condition=Ready ic/eks-us-east --timeout=20m +``` + +## Cache the weights + +A gang reads its weights from a shared cache, so pods don't each pull a copy. +Create the namespace and the cache: + +```bash +kubectl create namespace ml-team +``` + +{{< manifests "guides/serving-multi-node-on-dynamo/model-cache.yaml" >}} + +## Deploy the gang + +The `Leader` and `Worker` run the same `vllm serve`, differing only in node rank. +`$(MODELPLANE_LEADER_ADDRESS)` resolves to the leader on either stack, but +`$(MODELPLANE_RANK)` isn't injected on Dynamo yet, so the worker derives its rank +from Grove's `GROVE_PCLQ_POD_INDEX`. +[Multi-node deployments]({{< ref "/models/model-deployment.md#multi-node" >}}) +covers this. Both load with `--load-format modelexpress`, so they read the cached +weights through the Dynamo stack's ModelExpress server. + +{{< manifests "guides/serving-multi-node-on-dynamo/model-deployment.yaml" >}} + +Wait until `READY` shows `True`. The first start hydrates the cache, so it's +slower than later ones: + +```bash +kubectl get md -n ml-team --watch +``` + +On the workload cluster the gang is a Grove `PodCliqueSet`, the Dynamo stack's +multi-node workload in place of a LeaderWorkerSet: + +```bash +kubectl get podcliquesets.grove.io -A # workload cluster +``` + +## Expose and query + +{{< manifests "guides/serving-multi-node-on-dynamo/model-service.yaml" >}} + +Read the endpoint's address and send it a request. The `model` field is the +`--served-model-name` the deployment sets: + +```bash +ADDRESS=$(kubectl get ms qwen2-5-14b -n ml-team -o jsonpath='{.status.address}') +kubectl run -i --rm curl-test \ + --image=curlimages/curl \ + --restart=Never \ + --env="ADDRESS=$ADDRESS" \ + -- sh -c 'curl -s "$ADDRESS/v1/chat/completions" \ + -H "Content-Type: application/json" \ + -d "{\"model\":\"qwen2.5-14b\",\"messages\":[{\"role\":\"user\",\"content\":\"What is Kubernetes in one sentence?\"}],\"max_tokens\":100}"' +``` + +The request routes through the gateway to the leader, which serves the gang's one +endpoint. + +## Scale out with peer-to-peer loading + +Add a second replica and ModelExpress shows what it's for. The first replica +seeds its weights from the cache and publishes itself as a source; the second +loads them straight from the first, peer-to-peer, rather than reading the cache +again. + +Each replica is a gang of two nodes, so a second replica needs two more nodes. +Grow the pool to four, then scale the deployment: + +```bash +kubectl patch ic/eks-us-east --type=json -p '[ + {"op":"replace","path":"/spec/nodePools/0/nodeCount","value":4}, + {"op":"replace","path":"/spec/nodePools/0/minNodeCount","value":4}, + {"op":"replace","path":"/spec/nodePools/0/maxNodeCount","value":4}]' +kubectl patch md/qwen2-5-14b -n ml-team --type=merge -p '{"spec":{"replicas":2}}' +``` + +Once the second gang starts, watch its leader load from the first over +ModelExpress: + +```bash +kubectl logs -n default -l modelplane.ai/clique-role=leader -c engine --tail=-1 \ + | grep "source worker" # workload cluster +# [Worker 0] Trying source worker 63d91022 (266 tensors) +``` + diff --git a/docs/manifests/guides/serving-multi-node-on-dynamo/inference-class.yaml b/docs/manifests/guides/serving-multi-node-on-dynamo/inference-class.yaml new file mode 100644 index 000000000..52df767a5 --- /dev/null +++ b/docs/manifests/guides/serving-multi-node-on-dynamo/inference-class.yaml @@ -0,0 +1,27 @@ +# EKS g6.8xlarge, one NVIDIA L4 per node. A single L4 can't hold the 14B model, +# so the gang spans two nodes. This node has enough memory to hold the weights as +# they load, and the 100 GB disk holds the vLLM image. +apiVersion: modelplane.ai/v1alpha1 +kind: InferenceClass +metadata: + name: l4-1x-g6 +spec: + description: "EKS g6.8xlarge, 1x NVIDIA L4" + provisioning: + provider: EKS + eks: + instanceType: g6.8xlarge + diskSizeGb: 100 + accelerator: + type: nvidia-l4 + count: 1 + devices: + - name: gpu + claim: DRA + driver: gpu.nvidia.com + deviceClassName: gpu.nvidia.com + count: 1 + attributes: + architecture: { string: Ada Lovelace } + capacity: + memory: { value: "23034Mi" } # L4's real reported VRAM (not the nominal 24GB) diff --git a/docs/manifests/guides/serving-multi-node-on-dynamo/inference-cluster.yaml b/docs/manifests/guides/serving-multi-node-on-dynamo/inference-cluster.yaml new file mode 100644 index 000000000..32a281063 --- /dev/null +++ b/docs/manifests/guides/serving-multi-node-on-dynamo/inference-cluster.yaml @@ -0,0 +1,25 @@ +# An EKS cluster running the Dynamo serving stack, with a two-node L4 pool so a +# gang can span both nodes. spec.stack: Dynamo installs Grove and the KAI +# Scheduler, which gang-schedule the leader and worker together and compose them +# as a Grove PodCliqueSet, and ModelExpress, which serves cached weights to the +# gang and moves them between replicas peer-to-peer. +apiVersion: modelplane.ai/v1alpha1 +kind: InferenceCluster +metadata: + name: eks-us-east + labels: + modelplane.ai/region: us-east +spec: + stack: Dynamo + cluster: + source: EKS + eks: + region: us-east-1 + nodePools: + - name: gpu-l4 + className: l4-1x-g6 + nodeCount: 2 + minNodeCount: 2 + maxNodeCount: 2 + zones: + - us-east-1b diff --git a/docs/manifests/guides/serving-multi-node-on-dynamo/model-cache.yaml b/docs/manifests/guides/serving-multi-node-on-dynamo/model-cache.yaml new file mode 100644 index 000000000..bbb726e96 --- /dev/null +++ b/docs/manifests/guides/serving-multi-node-on-dynamo/model-cache.yaml @@ -0,0 +1,14 @@ +# The shared read-write-many cache the gang serves from, hydrated once from +# Hugging Face. Both gang pods mount it and read weights from it over EFS, +# instead of each pulling its own copy. Qwen2.5-14B is open, so it needs no +# token. Its FP16 weights are about 29 GB, so sizeGiB leaves headroom. +apiVersion: modelplane.ai/v1alpha1 +kind: ModelCache +metadata: + name: qwen2-5-14b + namespace: ml-team +spec: + source: HuggingFace + huggingFace: + repo: Qwen/Qwen2.5-14B-Instruct + sizeGiB: 40 diff --git a/docs/manifests/guides/serving-multi-node-on-dynamo/model-deployment.yaml b/docs/manifests/guides/serving-multi-node-on-dynamo/model-deployment.yaml new file mode 100644 index 000000000..5399d4a76 --- /dev/null +++ b/docs/manifests/guides/serving-multi-node-on-dynamo/model-deployment.yaml @@ -0,0 +1,101 @@ +# Qwen2.5-14B served across two L4 nodes as a gang. The FP16 weights (~29 GB) +# don't fit one L4's 23 GB, so the engine is a Leader + Worker gang, +# pipeline-parallel across two g6.8xlarge nodes with one L4 each. Both pods mount +# the shared ModelCache. +# +# The cluster runs the Dynamo stack, so Grove and the KAI Scheduler gang-schedule +# the two pods, and Modelplane composes them as a Grove PodCliqueSet. +# $(MODELPLANE_LEADER_ADDRESS) resolves to the leader on Dynamo, but +# $(MODELPLANE_RANK) isn't injected there yet (modelplaneai/modelplane#418), so +# each command sets its own --node-rank: 0 on the leader, and +# $$((GROVE_PCLQ_POD_INDEX + 1)) on the worker ($$ escapes past Kubernetes, +# leaving $((...)) for the shell to evaluate). +# +# Notes on the engine flags: +# --pipeline-parallel-size=2 splits the model across the two nodes; +# --tensor-parallel-size=1 keeps one GPU per node. Pipeline parallelism sends +# only activations between nodes, so it stays light on the network. +# --distributed-executor-backend=mp is vLLM's native multiprocessing multi-node +# path; vllm/vllm-openai:v0.23.0 no longer ships Ray. +# --load-format modelexpress loads weights through the ModelExpress server the +# Dynamo stack runs. The first replica seeds from the cache and publishes +# itself; later replicas pull their weights peer-to-peer from a replica that +# already has them. The vLLM image doesn't ship the loader, so pip install it +# first. --load-format=runai_streamer is the alternative that always reads the +# cache directly, on any stack. +# --max-model-len=8192 caps context so the KV cache fits alongside the weights. +apiVersion: modelplane.ai/v1alpha1 +kind: ModelDeployment +metadata: + name: qwen2-5-14b + namespace: ml-team +spec: + replicas: 1 + template: + spec: + modelCacheRef: + name: qwen2-5-14b + engines: + - name: qwen + members: + - role: Leader + nodeSelector: + devices: + - name: gpu + count: 1 + selectors: + - cel: | + device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("20Gi")) >= 0 + template: + spec: + containers: + - name: engine + image: vllm/vllm-openai:v0.23.0 + command: + - /bin/sh + - -c + - >- + pip install --index-url https://pypi.nvidia.com modelexpress && + exec vllm serve Qwen/Qwen2.5-14B-Instruct + --served-model-name=qwen2.5-14b + --tensor-parallel-size=1 + --pipeline-parallel-size=2 + --distributed-executor-backend=mp + --nnodes=2 --node-rank=0 + --master-addr=$(MODELPLANE_LEADER_ADDRESS) + --load-format modelexpress + --max-model-len=8192 + --gpu-memory-utilization=0.90 + --port=8000 + - role: Worker + worker: + nodes: 1 + nodeSelector: + devices: + - name: gpu + count: 1 + selectors: + - cel: | + device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("20Gi")) >= 0 + template: + spec: + containers: + - name: engine + image: vllm/vllm-openai:v0.23.0 + command: + - /bin/sh + - -c + - >- + pip install --index-url https://pypi.nvidia.com modelexpress && + exec vllm serve Qwen/Qwen2.5-14B-Instruct + --served-model-name=qwen2.5-14b + --tensor-parallel-size=1 + --pipeline-parallel-size=2 + --distributed-executor-backend=mp + --nnodes=2 --node-rank=$$((GROVE_PCLQ_POD_INDEX + 1)) + --master-addr=$(MODELPLANE_LEADER_ADDRESS) + --headless + --load-format modelexpress + --max-model-len=8192 + --gpu-memory-utilization=0.90 + --port=8000 diff --git a/docs/manifests/guides/serving-multi-node-on-dynamo/model-service.yaml b/docs/manifests/guides/serving-multi-node-on-dynamo/model-service.yaml new file mode 100644 index 000000000..1f9a3e61b --- /dev/null +++ b/docs/manifests/guides/serving-multi-node-on-dynamo/model-service.yaml @@ -0,0 +1,14 @@ +# Exposes the gang as one OpenAI-compatible URL. Modelplane composes one +# ModelEndpoint per replica, labeled modelplane.ai/deployment: qwen2-5-14b, so +# this selector reaches it. Read the public address from status.address: +# kubectl get ms qwen2-5-14b -n ml-team -o jsonpath='{.status.address}' +apiVersion: modelplane.ai/v1alpha1 +kind: ModelService +metadata: + name: qwen2-5-14b + namespace: ml-team +spec: + endpoints: + - selector: + matchLabels: + modelplane.ai/deployment: qwen2-5-14b diff --git a/docs/utils/vale/styles/config/vocabularies/Modelplane/accept.txt b/docs/utils/vale/styles/config/vocabularies/Modelplane/accept.txt index f70123286..d30e35332 100644 --- a/docs/utils/vale/styles/config/vocabularies/Modelplane/accept.txt +++ b/docs/utils/vale/styles/config/vocabularies/Modelplane/accept.txt @@ -251,6 +251,7 @@ H100 H200 EFA BF16 +FP16 FP8 INT4 AWQ @@ -530,6 +531,8 @@ all-or-nothing [Pp]eer-to-peer [Gg]ang-schedules? routable +pipeline-parallel +Qwen2.5-14B agent requirement