RoCE vs. InfiniBand for multi-node GPU training: when the fabric choice matters

Most of the hard decisions about GPU cluster networking get made before you run a single training step: how you map parallelism onto nodes, which fabric ties the nodes together, how big the cluster is and how you orchestrate it. Get those right and scaling is uneventful. Get them wrong and you either pay for bandwidth you never saturate or you starve expensive GPUs waiting on the network.

This is a decision framework, not a deployment tutorial. On a managed provider you usually take the topology you are given, so treat it as how to size and choose a cluster, not how to wire a switch. The vendors differ; the tradeoffs do not.

Node shape: eight GPUs per node is a constant, not a choice

Modern training nodes ship with eight GPUs. HGX-class H100, H200 and B200 machines put eight GPUs on one baseboard, and on Runpod Clusters that is the only node shape offered.

Inside that node, GPUs talk over NVLink. That path moves hundreds of GB/s per GPU on current hardware, fast enough that you can mostly treat an 8-GPU node as one large accelerator. The moment traffic crosses to another node it drops onto the network, which even at the top tier is roughly an order of magnitude slower per GPU than NVLink. If you have been comparing NVLink vs InfiniBand, they are not competitors: NVLink is the intra-node backplane, InfiniBand or RoCE is the inter-node fabric, and every multi-node cluster uses both.

Since the node shape is fixed, the real decision is how you map parallelism onto it. Keep the chattiest dimension, usually tensor parallelism, inside the node where NVLink absorbs it, and put data or pipeline parallelism across nodes where traffic is lighter and overlaps better with compute. What still varies between providers is the topology behind those eight GPUs: how many network interface cards (NICs) serve them, and whether the layout is rail-optimized so each GPU reaches its NIC without crossing a PCIe root. Verify the topology instead of assuming it.

Interconnect: RDMA is the entry fee, the fabric is the choice

Multi-node training requires an RDMA (Remote Direct Memory Access) fabric, either InfiniBand or RoCE. Plain TCP over Ethernet is not a cheaper tier; without RDMA, every gradient exchange drags the CPU into the transfer path and multi-node training slows to the point of being unusable.

That leaves two fabrics, and on current high-end training clusters they run at the same per-node bandwidth:

• InfiniBand, up to roughly 3,200 Gbps (3.2 Tbps) per node, with native RDMA and a centrally managed fabric.
• RoCE (RDMA over Converged Ethernet), routable in its current RoCEv2 form, which delivers up to the same 3.2 Tbps per node on the same NICs on current flagship SKUs, carrying RDMA over standard Ethernet switching.

InfiniBand vs RoCE

Since the bandwidth is identical, the differences are operational, and they only start to matter as the cluster grows.

1. InfiniBand ships with a centrally managed fabric, credit-based flow control and adaptive routing, which is why it tends to deliver the most predictable tail latency as collective sizes grow.
2. RoCEv2 carries RDMA over Ethernet switching and relies on explicit congestion notification (ECN) and priority flow control (PFC) to give RDMA the lossless behavior it expects. With that tuning done well, RoCEv2 has closed most of the operational gap, and most large datacenter operators have it dialed in.

The practical rule, from the engineers who operate Runpod's cluster fabrics: below roughly 32 nodes the two are hard to tell apart, so take whichever your provider gives you. At 32 nodes and beyond, prefer InfiniBand, because that is the scale where fabric management and congestion behavior separate the two, not raw speed.

That rule is an operational default, and it is aging in RoCE's favor. Meta trained Llama 3 on a 24,000-GPU RoCE cluster, and that result is less exceptional every year as ECN and PFC tuning matures across the industry. As a tenant on a managed cluster you do not own that tuning, so the useful question for your provider is not which fabric but whether production-hardened congestion tuning stands behind their RoCE tier. The default leans InfiniBand at scale because it removes that question, not because RoCE cannot get there. Treat the rule as your starting prior and let your own measurements overrule it. The rule also has a practical boundary: clusters past roughly the 8-node mark stop being a console checkout and become a capacity conversation with your provider, and at 32-plus nodes the fabric question belongs there. Below the self-serve ceiling it is already answered.

Cost and availability often settle the question before any of this does. InfiniBand is effectively a single-vendor ecosystem, and as of this writing lead times on InfiniBand switching for larger deployments run around 30 weeks. RoCE runs on commodity Ethernet hardware from multiple vendors, which prices lower per port and ships on normal timelines. You will rarely place that hardware order yourself, but the economics reach you anyway: fabrics built on Ethernet come online sooner and cost the provider less, and both show up in what you pay and in when capacity exists at all. A provider offering a RoCE tier is usually making a cost and availability call, not a performance concession.

The scaling behavior nobody puts on the pricing page

At 32 or more nodes, the selection of network fabric becomes critical, as different fabrics handle the growth of communication overhead differently depending on the chosen parallelism strategy. A well-implemented ring or tree all-reduce, the backbone of data-parallel training, keeps the bandwidth each GPU needs roughly constant as you add nodes. That is the point of the ring algorithm. All-to-all communication is the opposite: it shows up in mixture-of-experts (MoE) routing and some tensor- and pipeline-parallel schemes, and both per-GPU traffic and the number of simultaneous flows climb steeply with participant count.

So the sizing question is what your job puts on the wire, not how many nodes you rent.

An illustrative crossover (run your own numbers)

Train a model data-parallel across 8 nodes. All-reduce demand per GPU stays roughly flat, sits comfortably under the 3.2 Tbps per-node line rate, and either fabric keeps every GPU busy. At this scale the InfiniBand-vs-RoCE decision is close to invisible, which is what the 32-node rule predicts. Now switch to an MoE model with heavy all-to-all across 40 nodes. Flow count multiplies with participant count, traffic arrives in bursts, and the limiter becomes congestion and tail latency rather than line rate, because every collective finishes at the pace of its slowest participant. InfiniBand's managed fabric keeps the latency distribution tight while an untuned RoCE deployment develops stragglers, and GPUs sit idle waiting on gradients. There is no magic node count. Measure your job's achieved per-GPU comms bandwidth and MFU (model FLOPs utilization) first, then choose the fabric that keeps the GPUs busy at your target scale; the next section shows how. Fabric consistency you never stress is invisible; fabric inconsistency at scale idles the most expensive thing you own.

If you run dense fine-tunes on 2-8 nodes, you can skip the fabric comparison; either fabric is fine at your scale. The sections that matter for you are the measurement and NCCL sections that follow.

How to measure before you buy the next node

Four checks tell you whether the network or the math is your bottleneck, and none of them needs switch-side access, which matters because on a rented cluster you cannot log into the fabric.

1. Run your actual job at two scales first, because that answers the money question directly.
2. Train a fixed number of steps at your current node count and again at double it, and compare tokens per second per GPU.
3. If per-GPU throughput holds roughly flat, you are compute-bound and the bigger cluster is buying you wall-clock time. If it sags, communication is absorbing the new nodes, and paying for more of them buys you a bigger bill rather than a faster run.
4. Then isolate the network with nccl-tests, NVIDIA's standard collective benchmark. Run all_reduce_perf for data-parallel jobs and alltoall_perf for MoE-style jobs, one rank per GPU across every node. Build with MPI=1 for multi-node runs, and place the ranks explicitly so the test actually crosses the fabric. For two 8-GPU nodes under MPI:

mpirun -np 16 -H node1:8,node2:8 ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1

Without the host placement, all 16 ranks land on one node and you benchmark NVLink while believing you measured the network. Slurm users can substitute srun, and on Runpod Clusters the node addresses are already set in the environment.

Read the busbw column at the large message sizes and compare it against line rate in matching units: a 400 Gbps NIC moves 50 GB/s, so busbw in the low single digits of GB/s on that hardware is a path problem such as TCP fallback or the wrong interface, not a tuning problem. The nccl-tests performance doc explains how busbw is derived.

For the topology check, nvidia-smi topo -m prints the GPU-to-NIC affinity matrix from inside your container with no privileged access. GPUs that reach their NIC through a single PCIe bridge show PIX; a GPU that crosses the CPU interconnect to reach a NIC shows SYS and will underperform on that rail.

For MFU, estimate achieved FLOPs per second as 6 times parameter count times tokens per second for a dense transformer, then divide by aggregate peak FLOPs at your training precision from the GPU datasheet. The 6ND approximation undercounts attention, so treat this as the one soft number in the kit. It still moves in the right direction: if MFU drops when you add nodes, the network is where it went.

To put dollars on it: a 64-GPU job at $2.50 per GPU-hour costs $160 an hour. If GPUs stall on communication for 30% of each step, $48 of every hour buys nothing, which is roughly $860 over an 18-hour run, on every run, until someone reads the NCCL log. 

Software topology: making NCCL actually saturate the fabric

Hardware sets the ceiling. NCCL decides how close you get, and it is where most "our expensive network isn't helping" problems actually live.

The most common failure is NCCL silently falling back to TCP instead of using the RDMA path, so you pay for a 3.2 Tbps fabric and run at a fraction of it. Confirm the path before anything else with these NCCL environment variables:

• NCCL_DEBUG=INFO: read the init log. It prints which transport and which HCA each rank selected. Sockets or TCP where you expected the RDMA path is your bug.
• NCCL_IB_HCA: pin NCCL to the right InfiniBand device(s) instead of letting it guess.
• NCCL_SOCKET_IFNAME: point the bootstrap and control path at the correct interface. Getting this wrong is a classic cause of hangs and slow rendezvous.
• NCCL_IB_DISABLE=0: make sure an inherited environment has not disabled the RDMA path.

If you are running on Runpod Clusters, skip the guesswork: the cluster configuration reference documents exactly which interfaces and environment variables the platform pre-sets, including the interface NCCL_SOCKET_IFNAME should point at and why the external interface must not carry inter-node traffic.

When a multi-node all-reduce runs at, say, a third of line rate, work in this order: confirm with NCCL_DEBUG=INFO that it took the RDMA path and not TCP; if the path is right, check NIC-to-GPU affinity and rail alignment, because a GPU talking to a NIC across the wrong PCIe root will underperform; then confirm the fabric itself with ibv_devinfo (link up at the expected rate), mlxlink and ibqueryerrors (look for symbol and link errors, not just "active"). Suspect your training code last.

For orchestration, the common paths are distributed PyTorch (torchrun with DDP or FSDP) and Slurm for HPC-style scheduling. Managed clusters usually pre-wire the rank, world-size and master-address variables they expect, so you are not hand-assembling the launch topology.

The framework, in one pass

1. Start from your workload's communication pattern: data-parallel all-reduce or all-to-all-heavy.
2. Measure before you scale: the two-scale throughput test and nccl-tests tell you whether the network or the math is your bottleneck.
3. Treat RDMA as mandatory for any multi-node job. At 32 nodes and beyond, default to Infiniband, ask what congestion tuning stands behind a RoCE tier, and let price, availability and your measurements overrule the default.
4. Map parallelism so the heaviest traffic stays on NVLink inside the node, and verify NIC-to-GPU topology with nvidia-smi topo -m rather than assuming it.
5. Pick orchestration (PyTorch or Slurm) to match your team.
6. Validate with NCCL_DEBUG=INFO, nccl-tests and the fabric tools before blaming your model code.

Right-size the fabric to the job, not to a round number of nodes. That is the difference between a cluster that scales and a bill nobody can explain.

Getting started

Runpod Clusters run the RDMA fabrics this post describes, and the configuration reference documents the interfaces and environment variables up front, so the debugging section above starts from a known topology instead of a guess. Deploy from the console in minutes.

1. Clusters overview: runpod.io/product/clusters
2. Docs: docs.runpod.io/instant-clusters
3. PyTorch on Clusters: docs.runpod.io/instant-clusters/pytorch
4. Slurm on Clusters: docs.runpod.io/instant-clusters/slurm
5. Discord: join us here

‍