Skip to main contentBlog
Optimize multi-node GPU cluster performance and cost-efficiency by aligning parallelism strategies, network fabric selection, and performance benchmarking.
Most of the hard decisions about GPU cluster networking get made before you run a single training step: how you map parallelism onto nodes, which fabric ties the nodes together, how big the cluster is and how you orchestrate it. Get those right and scaling is uneventful. Get them wrong and you either pay for bandwidth you never saturate or you starve expensive GPUs waiting on the network.
This is a decision framework, not a deployment tutorial. On a managed provider you usually take the topology you are given, so treat it as how to size and choose a cluster, not how to wire a switch. The vendors differ; the tradeoffs do not.
Modern training nodes ship with eight GPUs. HGX-class H100, H200 and B200 machines put eight GPUs on one baseboard, and on Runpod Clusters that is the only node shape offered.
Inside that node, GPUs talk over NVLink. That path moves hundreds of GB/s per GPU on current hardware, fast enough that you can mostly treat an 8-GPU node as one large accelerator. The moment traffic crosses to another node it drops onto the network, which even at the top tier is roughly an order of magnitude slower per GPU than NVLink. If you have been comparing NVLink vs InfiniBand, they are not competitors: NVLink is the intra-node backplane, InfiniBand or RoCE is the inter-node fabric, and every multi-node cluster uses both.
Since the node shape is fixed, the real decision is how you map parallelism onto it. Keep the chattiest dimension, usually tensor parallelism, inside the node where NVLink absorbs it, and put data or pipeline parallelism across nodes where traffic is lighter and overlaps better with compute. What still varies between providers is the topology behind those eight GPUs: how many network interface cards (NICs) serve them, and whether the layout is rail-optimized so each GPU reaches its NIC without crossing a PCIe root. Verify the topology instead of assuming it.
Multi-node training requires an RDMA (Remote Direct Memory Access) fabric, either InfiniBand or RoCE. Plain TCP over Ethernet is not a cheaper tier; without RDMA, every gradient exchange drags the CPU into the transfer path and multi-node training slows to the point of being unusable.
That leaves two fabrics, and on current high-end training clusters they run at the same per-node bandwidth:
Since the bandwidth is identical, the differences are operational, and they only start to matter as the cluster grows.
The practical rule, from the engineers who operate Runpod's cluster fabrics: below roughly 32 nodes the two are hard to tell apart, so take whichever your provider gives you. At 32 nodes and beyond, prefer InfiniBand, because that is the scale where fabric management and congestion behavior separate the two, not raw speed.
That rule is an operational default. Meta trained Llama 3 on a 24,000-GPU RoCE cluster, which shows what tuned RoCE does when a dedicated network team owns the congestion tuning. As a tenant on a managed cluster you do not own that tuning, and that is why the default leans InfiniBand as participant counts grow. Treat the rule as your starting prior and let your own measurements overrule it. The rule also has a practical boundary: clusters past roughly the 8-node mark stop being a console checkout and become a capacity conversation with your provider, and at 32-plus nodes the fabric question belongs there. Below the self-serve ceiling it is already answered.
At 32 or more nodes, the selection of network fabric becomes critical, as different fabrics handle the growth of communication overhead differently depending on the chosen parallelism strategy. A well-implemented ring or tree all-reduce, the backbone of data-parallel training, keeps the bandwidth each GPU needs roughly constant as you add nodes. That is the point of the ring algorithm. All-to-all communication is the opposite: it shows up in mixture-of-experts (MoE) routing and some tensor- and pipeline-parallel schemes, and both per-GPU traffic and the number of simultaneous flows climb steeply with participant count.
So the sizing question is what your job puts on the wire, not how many nodes you rent.
Train a model data-parallel across 8 nodes. All-reduce demand per GPU stays roughly flat, sits comfortably under the 3.2 Tbps per-node line rate, and either fabric keeps every GPU busy. At this scale the InfiniBand-vs-RoCE decision is close to invisible, which is what the 32-node rule predicts. Now switch to an MoE model with heavy all-to-all across 40 nodes. Flow count multiplies with participant count, traffic arrives in bursts, and the limiter becomes congestion and tail latency rather than line rate, because every collective finishes at the pace of its slowest participant. InfiniBand's managed fabric keeps the latency distribution tight while an untuned RoCE deployment develops stragglers, and GPUs sit idle waiting on gradients. There is no magic node count. Measure your job's achieved per-GPU comms bandwidth and MFU (model FLOPs utilization) first, then choose the fabric that keeps the GPUs busy at your target scale; the next section shows how. Fabric consistency you never stress is invisible; fabric inconsistency at scale idles the most expensive thing you own.
If you run dense fine-tunes on 2-8 nodes, you can skip the fabric comparison; either fabric is fine at your scale. The sections that matter for you are the measurement and NCCL sections that follow.
Four checks tell you whether the network or the math is your bottleneck, and none of them needs switch-side access, which matters because on a rented cluster you cannot log into the fabric.
mpirun -np 16 -H node1:8,node2:8 ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1Without the host placement, all 16 ranks land on one node and you benchmark NVLink while believing you measured the network. Slurm users can substitute srun, and on Runpod Clusters the node addresses are already set in the environment.
Read the busbw column at the large message sizes and compare it against line rate in matching units: a 400 Gbps NIC moves 50 GB/s, so busbw in the low single digits of GB/s on that hardware is a path problem such as TCP fallback or the wrong interface, not a tuning problem. The nccl-tests performance doc explains how busbw is derived.
For the topology check, nvidia-smi topo -m prints the GPU-to-NIC affinity matrix from inside your container with no privileged access. GPUs that reach their NIC through a single PCIe bridge show PIX; a GPU that crosses the CPU interconnect to reach a NIC shows SYS and will underperform on that rail.
For MFU, estimate achieved FLOPs per second as 6 times parameter count times tokens per second for a dense transformer, then divide by aggregate peak FLOPs at your training precision from the GPU datasheet. The 6ND approximation undercounts attention, so treat this as the one soft number in the kit. It still moves in the right direction: if MFU drops when you add nodes, the network is where it went.
To put dollars on it: a 64-GPU job at $2.50 per GPU-hour costs $160 an hour. If GPUs stall on communication for 30% of each step, $48 of every hour buys nothing, which is roughly $860 over an 18-hour run, on every run, until someone reads the NCCL log.
Hardware sets the ceiling. NCCL decides how close you get, and it is where most "our expensive network isn't helping" problems actually live.
The most common failure is NCCL silently falling back to TCP instead of using the RDMA path, so you pay for a 3.2 Tbps fabric and run at a fraction of it. Confirm the path before anything else with these NCCL environment variables:
If you are running on Runpod Clusters, skip the guesswork: the cluster configuration reference documents exactly which interfaces and environment variables the platform pre-sets, including the interface NCCL_SOCKET_IFNAME should point at and why the external interface must not carry inter-node traffic.
When a multi-node all-reduce runs at, say, a third of line rate, work in this order: confirm with NCCL_DEBUG=INFO that it took the RDMA path and not TCP; if the path is right, check NIC-to-GPU affinity and rail alignment, because a GPU talking to a NIC across the wrong PCIe root will underperform; then confirm the fabric itself with ibv_devinfo (link up at the expected rate), mlxlink and ibqueryerrors (look for symbol and link errors, not just "active"). Suspect your training code last.
For orchestration, the common paths are distributed PyTorch (torchrun with DDP or FSDP) and Slurm for HPC-style scheduling. Managed clusters usually pre-wire the rank, world-size and master-address variables they expect, so you are not hand-assembling the launch topology.
Right-size the fabric to the job, not to a round number of nodes. That is the difference between a cluster that scales and a bill nobody can explain.
Runpod Clusters run the RDMA fabrics this post describes, and the configuration reference documents the interfaces and environment variables up front, so the debugging section above starts from a known topology instead of a guess. Deploy from the console in minutes.
Blog Posts
.jpeg)
