# Multi-node serving on Dynamo

Serve a model too large for one GPU across two nodes, gang-scheduled by the Dynamo stack.

Source: /guides/serving-multi-node-on-dynamo/

<!-- vale write-good.Passive = NO -->
Qwen2.5-14B's FP16 weights are about 29 GB, larger than one NVIDIA L4's 23 GB, so
it serves across two nodes as a gang: a `Leader` and a `Worker`, one L4 each,
pipeline-parallel across the pair over an EFA fabric. On a
[Dynamo cluster]({{< ref "/platform/inference-cluster.md#serving-stack" >}}) Grove
and the KAI Scheduler gang-schedule the two pods together, Modelplane composes them
as a Grove `PodCliqueSet`, and they load their weights peer-to-peer with NVIDIA
ModelExpress.

This is the [getting started tour]({{< ref "/getting-started" >}}) scaled to two
nodes: one larger model on a `spec.stack: Dynamo` cluster, with EFA and
ModelExpress for the gang.
[Set up the platform]({{< ref "/getting-started/build-the-platform.md" >}}) first,
for the gateway and cloud credentials, then apply the manifests below.

## Register a Dynamo cluster

The `InferenceClass` describes a single-L4 node, sized up from the getting started
tour's for the larger model's weights and with EFA for the fabric. The
`InferenceCluster` runs two of them, sets `spec.stack: Dynamo` so Modelplane
installs Grove and the KAI Scheduler, and sets `fabric: EFA` on the pool.

{{< manifests "guides/serving-multi-node-on-dynamo/inference-class.yaml" >}}

{{< manifests "guides/serving-multi-node-on-dynamo/inference-cluster.yaml" >}}

Provisioning the pool and installing the stack takes about 15 minutes:

```bash
kubectl wait --for=condition=Ready ic/eks-us-east --timeout=20m
```

## Cache the weights

A gang reads its weights from a shared cache, so pods don't each pull a copy.
Create the namespace and the cache:

```bash
kubectl create namespace ml-team
```

{{< manifests "guides/serving-multi-node-on-dynamo/model-cache.yaml" >}}

## Deploy the gang

The `Leader` and `Worker` run the same `vllm serve`, differing only in node rank.
`$(MODELPLANE_LEADER_ADDRESS)` resolves to the leader on either stack, but
`$(MODELPLANE_RANK)` isn't injected on Dynamo yet, so the worker derives its rank
from Grove's `GROVE_PCLQ_POD_INDEX`.
[Multi-node deployments]({{< ref "/models/model-deployment.md#multi-node" >}})
covers this. Both opt into ModelExpress with `--load-format modelexpress`, so the
worker pulls its weights from the leader over EFA rather than reading the cache
again.

{{< manifests "guides/serving-multi-node-on-dynamo/model-deployment.yaml" >}}

Wait until `READY` shows `True`. The first start hydrates the cache, so it's
slower than later ones:

```bash
kubectl get md -n ml-team --watch
```

On the workload cluster the gang is a Grove `PodCliqueSet`, the Dynamo stack's
multi-node workload in place of a LeaderWorkerSet:

```bash
kubectl get podcliquesets.grove.io -A   # workload cluster
```

## Expose and query

{{< manifests "guides/serving-multi-node-on-dynamo/model-service.yaml" >}}

Read the endpoint's address and send it a request. The `model` field is the
`--served-model-name` the deployment sets:

```bash
ADDRESS=$(kubectl get ms qwen2-5-14b -n ml-team -o jsonpath='{.status.address}')
kubectl run -i --rm curl-test \
  --image=curlimages/curl \
  --restart=Never \
  --env="ADDRESS=$ADDRESS" \
  -- sh -c 'curl -s "$ADDRESS/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d "{\"model\":\"qwen2.5-14b\",\"messages\":[{\"role\":\"user\",\"content\":\"What is Kubernetes in one sentence?\"}],\"max_tokens\":100}"'
```

The request routes through the gateway to the leader, which serves the gang's one
endpoint.
<!-- vale write-good.Passive = YES -->