A CUDA out of memory error at the start of a training run is a decision point. You can reduce batch size until the model barely learns, or you can scale out. This guide covers the second option: distributed training across multiple GPUs on Runpod, written for teams who have already trained or fine-tuned on a single GPU and run into one of its ceilings. It runs from a single multi-GPU pod up to an Instant Cluster spanning 64 GPUs across eight nodes.
When you need multi-GPU training
Memory is usually what forces the decision, though it is not the only thing that does.
A 7B-parameter model in bfloat16 needs roughly 14 GB for its weights alone, before optimizer states, activations, or gradients. A 13B model needs around 26 GB. At 70B you are past what any single H100 can hold without aggressive offloading. Once the weights by themselves consume most of the card, the arithmetic stops working no matter how you tune the batch size.
Iteration speed matters just as much. When a single epoch stretches past a working day, you can no longer run enough experiments to learn anything from them, and distributing the work brings that cycle back into hours.
Dataset throughput is the third constraint, and it applies regardless of model size. For pretraining or large-scale fine-tuning across hundreds of millions of tokens, one GPU cannot feed itself fast enough. More GPUs mean more data workers reading and preprocessing in parallel.
Data parallelism, model parallelism, and pipeline parallelism
For most fine-tuning and mid-scale pretraining, data parallelism plus DeepSpeed ZeRO is the answer. The other strategies exist for the cases where it is not, and picking the wrong one costs days of debugging and idle GPUs.
Data parallelism replicates the full model on every GPU and splits the input batch across them. Each GPU computes gradients on its own shard of the batch, an all-reduce collective averages those gradients across the group, and every GPU then applies the same weight update. PyTorch DDP is the standard implementation, and when your model fits on one device it is all you need.
When the model no longer fits on one device, the split has to move into the model itself, putting different layers or tensor chunks on different GPUs. DeepSpeed and Megatron-LM both implement this, and model parallelism adds real engineering complexity and communication overhead compared to the data-parallel path.
Pipeline parallelism divides the model into sequential stages and feeds micro-batches through them, which keeps far more GPUs busy than naive model parallelism does. It pays for that with tuning: micro-batch size and pipeline depth both have to be right before the pipeline stops stalling.
PyTorch also ships its own sharding implementation. Fully Sharded Data Parallel shards parameters, gradients, and optimizer states much as ZeRO Stage 3 does, without pulling in a dependency outside torch.distributed. Teams that want to stay inside core PyTorch generally reach for FSDP. Teams that want configurable ZeRO stages, CPU or NVMe offload, and Hugging Face Trainer integration generally reach for DeepSpeed.
Start with data parallelism and escalate only when ZeRO Stage 3 or FSDP runs out of room.
PyTorch DDP on a Runpod multi-GPU pod
A Runpod GPU pod supports up to eight GPUs on a single node. That eight-GPU ceiling makes a single pod the natural starting point for fine-tuning or training models up to a few billion parameters.
Step 1: Deploy a multi-GPU pod
In the Runpod console, navigate to Pods and click + Deploy Pod. Select your GPU type (H100 or A100 for serious training) and choose how many GPUs you need. Select the Runpod PyTorch template from the Template Gallery, which ships with a matched PyTorch and CUDA build so the environment is ready without manual setup. Check the gallery for the current versions, since the template tracks upstream releases. Launch the pod.
Step 2: Adapt your training script for DDP
DDP needs the process group initialized with NCCL, the model moved onto the GPU this process owns, and the model wrapped.
import os
import torch
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.utils.data import DataLoader
from torch.utils.data.distributed import DistributedSampler
local_rank = int(os.environ["LOCAL_RANK"])
# Bind the process to its GPU BEFORE init. Skip this order and collectives fall back
# to GPU 0 for barriers, wasting memory there and risking a hang on a bad rank mapping.
torch.cuda.set_device(local_rank)
# NCCL is the backend for GPU-to-GPU collectives; gloo is CPU-only and much slower here.
dist.init_process_group(backend="nccl")
model = YourModel().to(local_rank)
model = DDP(model, device_ids=[local_rank])
# Retune this for the larger effective batch (see below); it is not a free carry-over.
optimizer = torch.optim.AdamW(model.parameters(), lr=3e-5)
loss_fn = torch.nn.CrossEntropyLoss()
# DistributedSampler gives each process a disjoint slice of the dataset.
sampler = DistributedSampler(dataset)
# pin_memory is what makes the non_blocking copies below actually asynchronous.
dataloader = DataLoader(dataset, sampler=sampler, batch_size=batch_size, pin_memory=True)
try:
for epoch in range(num_epochs):
# Without this, every process replays the same order each epoch.
sampler.set_epoch(epoch)
for inputs, targets in dataloader:
# Data lands on the CPU by default; move it to this rank's GPU.
inputs = inputs.to(local_rank, non_blocking=True)
targets = targets.to(local_rank, non_blocking=True)
# bf16 autocast skips the GradScaler bookkeeping that fp16 requires.
with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
loss = loss_fn(model(inputs), targets)
# backward() stays outside autocast; gradients are computed in fp32.
loss.backward()
optimizer.step()
optimizer.zero_grad(set_to_none=True)
finally:
# Releases NCCL communicators even if training raises.
dist.destroy_process_group()Call sampler.set_epoch(epoch) at the start of every epoch. Skip it and every process sees the same data order on each pass, which degrades training quality without throwing an error.
Your effective batch size just multiplied by the number of ranks, so the learning rate you tuned on one GPU is now being applied to eight times as much data per step. Resist reaching for a formula here. The linear scaling rule that gets quoted most often comes from large-batch SGD work; Adam and AdamW conventionally use square-root scaling instead, and recent work suggests even that overshoots at large batch sizes. Retune it with a short warmup and watch the loss curve rather than trusting a multiplier.
A second failure mode is quieter, because it produces no error at all. If part of your model yields no gradient on a given step, DDP waits on an all-reduce that never arrives and the job simply hangs. DistributedDataParallel(..., find_unused_parameters=True) handles that case, at the cost of an extra graph traversal on every step, so turn it on when you hit the hang and not before.
Step 3: Launch with torchrun
On a single eight-GPU pod:
torchrun --nproc_per_node=8 train.pytorchrun handles process spawning and sets LOCAL_RANK, RANK, and WORLD_SIZE for you, so none of that coordination belongs in your script.
DDP still replicates the whole model on every GPU, so the model has to fit on one card. Past that ceiling, DeepSpeed ZeRO is the next step.
DeepSpeed ZeRO stages
DeepSpeed’s Zero Redundancy Optimizer solves the memory problem DDP leaves open. Standard DDP keeps a full copy of weights, optimizer states, and gradients on every GPU. Communication stays cheap that way, but every GPU pays full price in memory for state it will never uniquely own. ZeRO partitions that state instead, in stages that trade communication for memory.
Stage 1 partitions the optimizer states, so each GPU keeps only its shard of them and delivers roughly a 4x reduction in optimizer memory per GPU. That is a large share of the card back for one config change, because optimizer state is one of the heaviest things resident during training.
Stage 2 adds gradient partitioning on top: after the backward pass, each GPU keeps only the gradients for the parameters it owns. This is where most fine-tuning workloads settle, because the communication cost is still modest and the memory saved is large.
Stage 3 goes further and partitions the weights themselves, which is how a 70B model trains across GPUs that could never individually hold it. Each GPU now reconstructs layers on demand, so the forward pass carries extra all-gather traffic.
ZeRO-Offload, available in Stage 2 and Stage 3, moves optimizer states and gradients out to CPU RAM. On an A100 80 GB pod that stretches single-GPU training to models approaching roughly 10B parameters before you need a second card at all.
DeepSpeed installs with pip and reads a JSON config:
# Pin it: the ZeRO config schema and CLI flags have shifted across releases.
pip install "deepspeed>=0.19,<0.20"{
"zero_optimization": {
"stage": 2,
"allgather_partitions": true,
"reduce_scatter": true,
"overlap_comm": true,
"offload_optimizer": {
"device": "cpu",
"pin_memory": true
}
},
"bf16": {
"enabled": true
},
"train_micro_batch_size_per_gpu": 4,
"gradient_accumulation_steps": 2
}Drop the offload_optimizer block if you are not offloading; for Stage 3 you can add offload_param alongside it. Note that DeepSpeed derives your effective global batch size from three values rather than taking it directly: train_micro_batch_size_per_gpu times gradient_accumulation_steps times the number of GPUs. The config above gives 4 x 2 x 8, or 64 samples per step on an eight-GPU pod. Change the GPU count without revisiting those keys and your effective batch size changes with it.
Launch with:
deepspeed --num_gpus=8 train.py --deepspeed --deepspeed_config ds_config.json--deepspeed is the boolean flag that turns DeepSpeed on; the config path travels separately on --deepspeed_config. Both are arguments to your script rather than the launcher, so call deepspeed.add_config_arguments(parser) on your argparse parser to accept them.
Hugging Face Trainer users pass the config path through TrainingArguments as deepspeed="ds_config.json" and the library handles the rest. One friction point comes with Stage 3, which shards checkpoints across ranks: the zero_to_fp32.py utility that ships with DeepSpeed consolidates those shards into a single file. Run it after the process group has shut down, not while other ranks are still writing.
One lever worth pulling before you add GPUs: activation checkpointing. Activations scale with batch size and sequence length rather than parameter count, and on long-context training they can rival the weights for memory. Checkpointing discards most of them during the forward pass and recomputes them during the backward, trading roughly 20 to 30 percent more compute for a large reduction in activation memory. PyTorch exposes it as torch.utils.checkpoint, DeepSpeed has its own activation-checkpointing config, and Hugging Face Trainer turns it on with gradient_checkpointing=True. It composes with every strategy in this guide, and it is often the difference between needing a second node and not.
Every technique to this point has been about fitting more work into eight GPUs. Past a certain size, no amount of partitioning or recomputation is enough, and the answer becomes more machines.
Moving to Runpod Instant Clusters
Instant Clusters provision two to eight nodes, up to 64 GPUs, in minutes. Crossing a node boundary changes where the GPUs talk to each other: inside a node they share a local bus, and on the SXM datacenter cards used for serious training that means NVLink (PCIe and consumer cards do not have it, so check against your chosen GPU type). Between nodes, every gradient synchronization crosses the network instead. Most of the setup below exists to make that crossing cheap.
Creating an Instant Cluster
In the Runpod console, navigate to the Instant Clusters page and click Create Cluster. You will configure:
- Cluster name: anything descriptive
- Pod count: number of nodes in the cluster
- GPU type: H100, H200, or B200 to get the full-speed inter-node fabric; A100 clusters run at half that bandwidth
- Pod Template: the Runpod PyTorch template, or your own custom Docker image
What Runpod pre-configures
Runpod sets the distributed environment variables automatically on every node, so there is no hostfile to write and no node addresses to coordinate by hand:
MASTER_ADDR/PRIMARY_ADDR: IP address of the primary nodeMASTER_PORT/PRIMARY_PORT: communication portNODE_RANK: this node’s rank (0 on the primary node)NUM_NODES: total number of nodes in the clusterNUM_TRAINERS: number of GPUs per nodeWORLD_SIZE: total GPUs across the whole cluster
PyTorch assigns LOCAL_RANK itself as it spawns each process.
Sharing the dataset and checkpoints
Every node needs to read the same dataset, and checkpoints need somewhere durable that outlives the cluster. Attach a network volume when you create the cluster and Runpod mounts it at /workspace on every node, backed by NVMe. Mounting the same volume on many nodes at once is supported and expected, so all eight nodes see one filesystem and you stage the dataset once rather than pulling a copy onto each node.
Two constraints come with it. A network volume lives in a single datacenter, which limits the cluster to wherever that volume already is, so create it in a region where your GPU type is actually available. And the NVMe sits on storage servers reached over the network rather than on the node itself, so read throughput is network-bound and shared across every node pulling from it. Size your dataloader workers and prefetch accordingly, and stage a hot subset onto local disk if the GPUs start waiting on data.
Starting the run on every node
Training does not start cluster-wide on its own. You open a web terminal on each pod and run the launch command there, once per node. On an eight-node cluster that is eight terminals, and every node needs the command before any node gets past init_process_group.
export NCCL_DEBUG=INFO
export NCCL_SOCKET_IFNAME=ens1
torchrun \
--nproc_per_node=$NUM_TRAINERS \
--nnodes=$NUM_NODES \
--node_rank=$NODE_RANK \
--master_addr=$MASTER_ADDR \
--master_port=$MASTER_PORT \
train.pyNCCL_SOCKET_IFNAME=ens1 tells NCCL which network interface carries inter-node traffic. Runpod exposes the high-speed interfaces as ens1 through ens8, running at up to 3200 Gbps on H100, H200, and B200 nodes and 1600 Gbps on A100. Point NCCL at one of those and you get full cross-node bandwidth. Leave it unset and NCCL may pick the external management interface, eth0, instead. That either throttles every gradient synchronization for the life of the run or fails outright with connection timeouts between nodes, which is the single most common way a multi-node job breaks on Runpod.
Debugging NCCL on multi-node
NCCL failures at multi-node scale are almost always transport, firewall, or a node that never reached the barrier: NCCL_SOCKET_IFNAME naming the wrong adapter, a firewall blocking the ports NCCL needs beyond its control port, or one node timing out before it reaches init_process_group. Running the first job with NCCL_DEBUG=INFO prints the transport path NCCL actually selected, which usually identifies the culprit on one read.
For runs measured in hours, checkpoint frequently to persistent storage. One node failing mid-run should not cost you the whole training job.
Save from rank 0 only. Every rank is running the same script, so an unguarded torch.save means all 64 processes write the same path simultaneously and you get a corrupt file rather than a checkpoint. Wrap the write in if dist.get_rank() == 0: and follow it with dist.barrier() so the other ranks wait for it to land before continuing.
Cost: multi-GPU pod vs. Instant Cluster
Cost on Runpod is GPU-hours times the per-GPU rate, billed by the second with no minimum spend and no long-term commitment.
That makes the comparison simpler than it first looks. Sixty-four GPUs for one hour and eight GPUs for eight hours are both 64 GPU-hours, so at the same per-GPU rate a cluster is not inherently the more expensive option. It buys the same compute in a fraction of the wall-clock time. What breaks the equivalence is scaling efficiency: a single pod keeps all eight GPUs on one node with no inter-node traffic, so utilization stays high, while across nodes every gradient synchronization crosses the network. Whatever efficiency you give up there is the real premium for finishing sooner, and it is why doubling the GPUs rarely halves the training time.
Put numbers on it. Fine-tuning a 13B model for 20 hours on an eight-GPU H100 SXM pod is 160 GPU-hours. At Secure Cloud’s $2.99 per GPU-hour that lands near $478, or about $430 on Community Cloud at $2.69. That $48 is not free, though: network volumes are a Secure Cloud feature, so a Community Cloud pod gives you no persistent shared storage and checkpoints have to land on the container disk or get pushed to an external store before the pod goes away. Run those same 160 GPU-hours as a two-node, 16-GPU cluster and the compute bill is comparable while the job finishes in roughly half the wall-clock time, less whatever you surrender to inter-node synchronization. Drop to A100 80 GB SXM at $1.49 and the same run costs around $238, which is the more consequential decision than pod-versus-cluster.
So the choice is rarely about the hourly rate. For most fine-tuning work, including large models under DeepSpeed ZeRO, an eight-GPU pod is hard to beat on cost per unit of work, and the card you pick moves the bill further than the pod-or-cluster question does. Rates move, so confirm against Runpod’s pricing page before you budget.
Whichever you land on, validate the training loop and DeepSpeed config on a single multi-GPU pod first. Debugging a distributed job is enough work across eight processes; across 64, on nodes you are paying for by the second, it is a different problem.
Frequently asked questions
How do I know the extra GPUs are actually helping?
Measure throughput, not utilization. nvidia-smi showing every card at 100% tells you the GPUs are busy, not that they are busy doing useful work: a badly configured job can spend most of its time waiting on gradient synchronization and still look saturated. Log samples per second across the whole run and compare it against your single-GPU baseline. Eight GPUs returning six to seven times the throughput is a healthy result. Three or four times means something is wrong, and the usual causes are a per-GPU batch too small to keep the card fed, a dataloader that cannot supply data fast enough, or NCCL quietly running over the wrong interface.
What happens if one node fails partway through a multi-node run?
The job stops. NCCL collectives are synchronous, so when one rank disappears the rest block until the operation times out and the run dies, usually with an NCCL error rather than a clean message. Nothing fails over automatically: you replace the node, relaunch on every node, and resume from the last checkpoint. That is what makes checkpoint frequency a decision about how much wall-clock time you can afford to lose, and why those checkpoints belong on a network volume rather than a node’s local disk.
Can I use Hugging Face Trainer with DeepSpeed ZeRO on Runpod?
Yes, and it carries to clusters without extra work. Pass the config path via deepspeed="ds_config.json" in TrainingArguments, and Trainer handles the distributed setup itself, reading the same WORLD_SIZE, NODE_RANK, and MASTER_ADDR values Runpod sets on every node. You still launch it once per node, and the Stage 3 checkpoint consolidation described above still applies.
Getting started
The whole decision reduces to one question: does the job fit in a single node’s memory and finish on a schedule you can live with? If it does, deploy a multi-GPU pod. If it does not, create an Instant Cluster.
The DDP initialization, DeepSpeed ZeRO config, and NCCL environment variables described here carry across every Runpod pod and cluster configuration, so the fine-tuning workflow you already run does not change as you scale it. If you got here from LoRA and QLoRA fine-tuning, none of that changes either. Same container, same model. You just have more GPUs behind it.
