Runpod Clusters Expansion: Scale your Clusters without recreating them

When you launch a distributed training or inference workload, you're making a bet about capacity.

Maybe 16 H100s are enough for the first run. Maybe the next experiment needs 24. Maybe your Slurm queue starts backing up, or your batch size changes, or the model you actually want to fine-tune needs more VRAM than the model you started testing with.

Historically, that meant deciding upfront how large your Cluster should be. If you guessed too small, you had to create a new Cluster, reconnect storage, reconfigure your runtime, and move the workload over.

Runpod Cluster Expansion changes that.

You can now add pods to an existing Cluster, while keeping the same GPU type, pod template, network storage, and private cluster network. The new pods provision into the Cluster and contribute additional GPUs, VRAM, and compute capacity without recreating the Cluster from scratch.

Note: Scaling isn't available for Clusters running on reserved or contracted hardware. To expand one of those, contact your account manager or the Runpod sales team.

What you'll do

In this walkthrough, we'll cover:

• When expanding a Cluster makes sense
• What changes under the hood when new pods are added
• How to expand a Cluster from the Runpod console
• How to verify the new capacity from Slurm or a distributed runtime
• What to watch for with NCCL, world size, and running jobs

Why expand instead of recreate?

Multi-node workloads are rarely static.

A few common examples:

• You start a distributed training test on two nodes, then scale to three or four once the input pipeline is stable.
• You run a smaller fine-tuning pass first, then increase global batch size for the real run.
• Your Slurm queue fills up and you want more workers without interrupting the existing Cluster.
• Your model or dataset changes, and the original GPU count is no longer enough.
• You want to validate cost and performance before committing to a larger reserved arrangement.

Cluster Expansion lets the Cluster grow in place. New pods join the Cluster's private network automatically and use the same GPU configuration as the existing nodes.

For example, a Cluster with two pods running 8x H100 SXM each starts with:

Pods: 2
GPUs: 16x H100 SXM
VRAM: 1280 GB

After adding one pod:

Pods: 3
GPUs: 24x H100 SXM
VRAM: 1920 GB

Your updated hourly cost is shown before you confirm the change.

Step 1: Start with a running Cluster

Cluster Expansion applies to an existing Cluster.

If your Cluster uses a private pool, the self-serve expansion flow is not available. Work with Runpod to adjust private-pool capacity instead.

In the Runpod console, go to:

Resources -> Clusters

Select the Cluster you want to expand.

Before scaling, check:

• The current pod count
• GPU type and region
• Whether the Cluster is running a managed Slurm configuration
• Whether any active training jobs assume a fixed node count
• Whether your workload can tolerate new capacity being added between runs

That last point matters. Many distributed training jobs, especially standard PyTorch DDP jobs, assume WORLD_SIZE is fixed when the process group starts. In that case, expanding the Cluster gives you more available capacity for the next job, not more GPUs inside a job that's already running.

Step 2: Add pods

From the Cluster page, click:

Scale Cluster

Under Additional pods, use the controls to choose how many pods to add.

Runpod shows a summary with the new total GPU count, total VRAM, and updated hourly cost. If everything looks right, click:

Scale this Cluster

The new pods begin provisioning immediately. Once they're running, they appear in the Cluster's pod list.

If the stepper doesn't allow the number of pods you need, that usually means available GPU inventory is the limiting factor. For larger capacity needs, use the Talk to our sales team option in the scaling dialog.

Step 3: Understand what changes

Each added pod matches the existing Cluster configuration:

• Same GPU type
• Same pod template
• Same network storage
• Same private Cluster network

Runpod Clusters also expose environment variables that distributed tools can use to discover the cluster topology:

echo $PRIMARY_ADDR
echo $PRIMARY_PORT
echo $NODE_ADDR
echo $NODE_RANK
echo $NUM_NODES
echo $NUM_TRAINERS
echo $WORLD_SIZE

For distributed training, the most important values are usually:

PRIMARY_ADDR / MASTER_ADDR: primary node address

PRIMARY_PORT / MASTER_PORT: rendezvous port

NODE_RANK: node rank

NUM_NODES: total node count

NUM_TRAINERS: GPUs per node

WORLD_SIZE: total GPUs across the Cluster

Don't assume a running distributed process will resize itself just because the Cluster now has more nodes. Most frameworks read topology at startup. If you're using PyTorch DDP, DeepSpeed, or similar tools, plan to restart or resubmit the job with the new world size.

Step 4: Verify networking

Clusters use dedicated high-bandwidth interfaces for inter-node communication. The management interface, eth0, is for external traffic and shouldn't be used for distributed GPU communication.

Start by confirming the new pods can find each other by hostname. If you scaled from two pods to three, node-0 and node-1 should be able to reach the new node, node-2, by name:

ping node-2

Run this from node-0 or node-1. A response means the new pod has joined the private Cluster network correctly. If it times out, check the pod's status in the console before you touch NCCL.

Runpod preconfigures the NCCL environment variables your Cluster needs — NCCL_SOCKET_IFNAME, NCCL_IB_HCA, and NCCL_IB_GID_INDEX — in /etc/nccl.conf. You shouldn't need to set these yourself.

If you're debugging a connection issue or running a nonstandard setup, override them directly:

export NCCL_DEBUG=INFO

If you see connection timeouts between nodes: check hostname resolution first, then confirm nothing in /etc/nccl.conf has been overridden.

Before you scale

A few operational details worth keeping in mind:

• Expansion depends on available GPU inventory at the time you scale.
• Added pods use the same configuration as the existing Cluster.
• Billing updates once the additional pods are running.
• To downsize, terminate the pods on the nodes you want to remove.
• For active distributed jobs, scale between runs unless your framework explicitly supports elastic training.
• If jobs are stuck pending in Slurm, check sinfo first to confirm the expected resources are actually available.

Why this matters

Clusters made multi-node GPU compute much easier to launch. Cluster Expansion makes it more forgiving to operate.

You no longer have to get the size exactly right on day one. Start with the Cluster that matches your current experiment, expand when the workload proves it needs more capacity, and keep the same environment as you move from test runs to larger jobs.

That's the real shift: less infrastructure reshuffling, more iteration on the work itself.

Cluster Expansion is generally available now. Expand your Cluster.

‍