Runpod Clusters connect 2 to 8 nodes (16 to 64 GPUs) at up to 3,200 Gbps per node on H100, H200 and B200 GPUs, billed by the second, and every node boots with the torchrun rendezvous variables already set. Any Runpod account can launch 2 nodes on demand. Up to 8 nodes takes a spend-limit increase, and clusters above 8 nodes (up to 512 GPUs) go through Runpod sales.
This guide takes a PyTorch training script that already runs on one 8-GPU node and runs it across two nodes. You launch the cluster, pin NCCL to the cluster network, run one torchrun command on each node, and confirm all 16 GPUs can see each other before you spend an hour on a real job.
TL;DR
- Runpod sets MASTER_ADDR, MASTER_PORT, NODE_RANK, NUM_NODES, NUM_TRAINERS and WORLD_SIZE on every node, and configures NCCL for RDMA, so the torchrun command needs no hard-coded addresses.
- Set NCCL_SOCKET_IFNAME=ens1 before you run anything. The eth0 interface on a Runpod node is for internet traffic only and causes NCCL timeouts.
- Run the same torchrun command on every node. Each node's own NODE_RANK does the rest.
- Prove the fabric works with a 16-GPU all-reduce before you start a training run.
What Runpod sets up before you run anything
Every node in a Runpod cluster starts with the variables a torchrun launch needs, as documented on the Clusters configuration page at docs.runpod.io/instant-clusters/configuration.
| Variable | What Runpod sets it to |
|---|---|
| PRIMARY_ADDR / MASTER_ADDR | The address of the primary node |
| PRIMARY_PORT / MASTER_PORT | The port of the primary node. All ports are available. |
| NODE_RANK | 0 on the primary node, then 1, 2 and so on |
| NUM_NODES | The number of nodes in the cluster |
| NUM_TRAINERS | The number of GPUs per node |
| WORLD_SIZE | NUM_NODES multiplied by NUM_TRAINERS |
| HOST_NODE_ADDR | PRIMARY_ADDR:PRIMARY_PORT |
Runpod also sets NCCL_IB_HCA and NCCL_IB_DISABLE=0 on each node (plus NCCL_IB_GID_INDEX on RoCE), so NCCL uses RDMA over the cluster fabric without extra configuration. On hosts without RDMA, NCCL falls back to TCP over the ens interfaces. torchrun sets LOCAL_RANK itself.
Step 1: launch a 2-node cluster
In the Runpod console, open the Clusters page and click Create Cluster. Keep Pod Count at 2, pick the 16x H100 SXM option, keep the Pod Template at its default (Runpod PyTorch), and click Deploy Cluster. If you want shared storage for checkpoints, click Network Volume during creation and attach a volume in the same region as the cluster. It mounts at /workspace on every node.
The same launch is one API call. Runpod's API v2 exposes POST /v2/clusters with compute.gpuTypeId, gpuCountPerPod and podCount, plus GET and DELETE on /v2/clusters/{id}. Set type to TRAINING for a torchrun job, or SLURM if your team submits with sbatch.
Step 2: pin NCCL to the cluster interface
Open a terminal on each node and run:
export NCCL_SOCKET_IFNAME=ens1
Runpod's docs are explicit about why: the 172.x addresses on eth0 are reserved for internet connectivity only and will result in connection timeouts for distributed workloads. While you are setting up, also turn on NCCL logging so a bad interface shows up in the first few lines of output instead of as a silent hang:
export NCCL_DEBUG=INFO
Step 3: run torchrun on every node
Run this on each node, from the directory that holds your script. The command is the one Runpod's PyTorch walkthrough uses, with the demo path swapped for your own.
torchrun --nproc_per_node=$NUM_TRAINERS --nnodes=$NUM_NODES --node_rank=$NODE_RANK --master_addr=$MASTER_ADDR --master_port=$MASTER_PORT train.py
Nothing in that line changes between nodes. On a 2-node H100 cluster it launches eight processes per node, one per GPU, and the node whose NODE_RANK is 0 hosts the rendezvous. Inside train.py, initialize with dist.init_process_group("nccl") and read LOCAL_RANK from the environment to pick the GPU; torchrun exports RANK, LOCAL_RANK, WORLD_SIZE, MASTER_ADDR and MASTER_PORT for you.
Step 4: check that 16 GPUs talk to each other
Before a real run, do one all-reduce across the whole cluster. Save this as allreduce_check.py on both nodes:
import os
import torch
import torch.distributed as dist
dist.init_process_group("nccl")
rank = dist.get_rank()
world = dist.get_world_size()
local_rank = int(os.environ["LOCAL_RANK"])
torch.cuda.set_device(local_rank)
t = torch.ones(1, device="cuda")
dist.all_reduce(t)
print(f"rank {rank}/{world} on node {os.environ['NODE_RANK']}: all_reduce sum = {int(t.item())}")
dist.destroy_process_group()
Run it with the Step 3 command, replacing train.py with allreduce_check.py. You should see 16 lines, one per rank, each ending in all_reduce sum = 16. A sum of 8 means the two nodes never joined the same group (check MASTER_ADDR and that you ran the command on both nodes). A hang with NCCL_DEBUG=INFO showing eth0 means Step 2 was skipped on one node.
Step 5: checkpoint to shared storage
Write checkpoints to /workspace when a network volume is attached. Every node sees the same files, so a restart on any node resumes from the same checkpoint, and the volume outlives the cluster. Runpod's docs note that the volume region must match the cluster region, and that Clusters cannot mount global volumes.
When something hangs
- Connection timeouts during init_process_group: NCCL_SOCKET_IFNAME is unset or points at eth0 on at least one node. Set it to ens1 everywhere.
- Workers cannot find the primary: use the PRIMARY_ADDR or MASTER_ADDR value Runpod set, never an address you copied from the console.
- One node is slower than the others: confirm both nodes are in the same data center and read the NCCL_DEBUG=INFO output for a TCP fallback where you expected RDMA.
- Cost keeps running after the job: delete the cluster. Runpod bills Clusters by the second with no minimum runtime, so an idle cluster is pure spend.
Clusters specs and prices
Prices as listed on runpod.io/pricing, updated November 4, 2026. H100 SXM and B200 Clusters are quoted through sales.
| GPU | Network per node | Nodes on demand | Max GPUs on demand | Price on runpod.io/pricing |
|---|---|---|---|---|
| H200 SXM | 3,200 Gbps | 2 to 8 | 64 | $6.35 per hour |
| H100 SXM | 3,200 Gbps | 2 to 8 | 64 | Contact sales |
| B200 | 3,200 Gbps | 2 to 8 | 64 | Contact sales |
| A100 SXM | 1,600 Gbps | 2 to 8 | 64 | $2.15 per hour |
Runpod's Clusters use InfiniBand or RoCE v2 depending on configuration, and reserved Clusters with 1-month to 12-month terms are priced through sales.
Scale past 8 nodes, or use Slurm
For clusters larger than 8 nodes, up to 512 GPUs, contact Runpod sales. If your team already submits jobs with sbatch, choose Slurm Cluster from the cluster type dropdown when you create the cluster; Runpod provisions the controller and workers, and the same network volume rules apply.
FAQ
How many nodes can I run on Runpod Clusters?
Runpod Clusters run 2 to 8 nodes (16 to 64 GPUs). Any account can launch 2 nodes on demand, up to 8 nodes takes a spend-limit increase, and clusters above 8 nodes (up to 512 GPUs) go through Runpod sales.
How fast is the network between nodes?
Runpod Clusters provide up to 3,200 Gbps per node on H100, H200 and B200, and 1,600 Gbps on A100, over InfiniBand or RoCE v2 depending on configuration.
How is a Runpod cluster billed?
Runpod bills Clusters by the second, with no minimum commitment, no minimum runtime and no termination fee.
Which NCCL setting do I need on Runpod?
Runpod's docs say to set NCCL_SOCKET_IFNAME=ens1 so NCCL traffic uses the cluster interface and not eth0, which is reserved for internet traffic.
Launch a 2-node cluster from the Runpod console and run the all-reduce check above; if the 16 lines come back, your training script is one torchrun command away.
