Renting H100 clusters on Runpod for distributed AI training comes down to three decisions you make before a single training step runs: which hardware variant, how the nodes talk to each other, and what stops the meter when a job dies.
This guide covers multi-node setup on Runpod's Instant Clusters end to end: choosing between PCIe, NVL, and SXM variants, understanding why inter-node topology changes your NCCL configuration, wiring up a multi-node torchrun or DeepSpeed launch that actually converges, and avoiding the $4,000 bill for a job that died three hours in and billed all weekend because you skipped auto-shutdown.
Written for practitioners who already know PyTorch distributed training, the focus here is which settings to configure and why they behave differently on Runpod's hardware. By the end you will have a multi-node cluster running a DeepSpeed or torchrun job, with NCCL tuned for Runpod's network topology.
Why H100 Clusters Now
An H100 SXM pod rents for roughly $2.99/GPU/hr on Runpod today, down from the $8+/GPU/hr many platforms charged in 2024. That price drop is what makes multi-node H100 training practical for labs that could not justify it a year ago, and it lands just as the tooling around it, DeepSpeed, FSDP, and Hugging Face Accelerate, has matured into years of production use.
At today's on-demand rate, a 2-node × 8-GPU cluster (16 GPUs) runs about $47.84/hr at pod rates, in the same ballpark as what many teams paid for A100 80GB clusters 18 months ago, though rates varied widely by provider. (Instant Cluster pricing is quoted per configuration rather than billed at the flat pod rate; more on that in Section 3.) The driver is straightforward: as Blackwell B200 availability increases and the teams chasing the newest hardware move up the stack, H100 capacity has loosened relative to demand.
For most LLM fine-tuning and mid-scale pre-training runs happening right now, H100 SXM5 is the sensible default: its NCCL and Transformer Engine support is mature and well-proven. Chasing B200 availability for a run that needs to ship next quarter is usually the wrong trade.
Choosing the Right H100 Variant for Distributed Training
Pick the wrong H100 form factor and communication overhead caps your scaling before compute ever does. Runpod offers three, and the spec that separates them for distributed training is inter-GPU bandwidth, which sets how far you can push tensor parallelism before all-reduce traffic dominates.
The H100 NVL pairs two H100 GPUs on a single PCIe card via an NVLink bridge, giving you 188 GB of combined HBM3. PyTorch sees two separate GPUs, but with NVLink bandwidth between them rather than PCIe. The pairing suits single-node large-model inference, and supervised fine-tuning where the model exceeds 80 GB VRAM and you would rather not pay the pipeline-parallelism tax to split it.
For multi-node distributed training, the SXM variant is generally the right choice. Its 900 GB/s bidirectional NVLink bandwidth within a node supports close-to-linear scaling with tensor parallelism, so you can usually run tensor-parallel up to 8 across all eight GPUs without all-reduce communication dominating (where the line sits depends on model size and batch configuration). PCIe inter-GPU bandwidth, by contrast, runs around 64 GB/s over the PCIe Gen5 bus, so tensor-parallel efficiency falls off fast, typically becoming the bottleneck past TP=2.
The PCIe variant is a reasonable fit for cost-sensitive fine-tuning runs using ZeRO-2 or ZeRO-3 with pure data parallelism, where inter-GPU bandwidth inside a node is not the critical path. If your batch is large enough that compute dominates communication, running PCIe on cheaper interruptible capacity trims cost without hurting throughput. (At on-demand rates the PCIe-vs-SXM gap is small, about $2.89 vs $2.99/GPU/hr; the savings come from the interruptible tier, not the variant.)
Variant decision summary:
- SXM5 (multi-node pre-training, tensor-parallel degree above 2, pipeline-parallel training)
- NVL (single-node fine-tuning of 13B–70B models where you need combined VRAM without sharding across nodes)
- PCIe (cost-sensitive data-parallel fine-tuning with frequent checkpointing that can tolerate restarts)
Pricing Breakdown
Two things drive what you pay: the GPU variant (Section 2) and how you deploy it. One catch up front: Instant Clusters, the multi-node model this guide uses, are priced by quote for H100 (GPU type × node count), not at a flat public rate. So the per-GPU figures below are the single-pod on-demand rates. Treat them as the proxy you size a cluster budget against, and confirm the actual cluster quote in the console. Current rates are on the Runpod pricing page.
Within the pod rates, the tier you pick trades cost against interruptibility: Community Cloud can evict your job, Secure Cloud carries an SLA, and Serverless does not fit multi-node training at all.
Community Cloud puts you on host-owner hardware, usually below the on-demand rate for the same card but host-dependent and variable. The catch is interruptibility: a host can reclaim its hardware, which means your job can be evicted. For runs that checkpoint frequently, say every few hundred steps, that is manageable. For a long pre-training run where losing several hours of compute hurts, it is a real risk. Restart logic here means detecting a fresh pod on boot, globbing the latest checkpoint off the network volume, and reloading model, optimizer, and scheduler state before resuming the step counter, so an eviction costs you a restart rather than the run.
Secure Cloud is Runpod-owned infrastructure with an SLA, billed at the on-demand catalog rate: roughly $2.89/hr for H100 PCIe, $2.99/hr for H100 SXM, and $3.19/hr for the 94 GB H100 NVL. This is the tier for production training runs with hard deadlines.
Serverless bills per second (H100 lists around $4.55/hr at full utilization) and is built for inference endpoints. The cold-start overhead and per-request model do not map to long-running, coordinated multi-node training.
Cost comparison for a 72-hour fine-tuning run on a 2-node × 8×H100 SXM cluster (at on-demand pod rates, since cluster pricing is quoted):
On-demand at $2.99/GPU/hr:
2 nodes × 8 GPUs × $2.99/GPU/hr × 72 hrs = $3,444.48Community Cloud H100 PCIe (interruptible, host-dependent) at a ~$2.00/hr low-end rate:
2 nodes × 8 GPUs × $2.00/GPU/hr × 72 hrs = $2,304.00
That is roughly an $1,140 delta, and whether it is worth the eviction risk depends on your checkpoint strategy. Checkpoint every 15 minutes and a restart costs you maybe 30 minutes of re-run. Checkpoint every six hours on a long custom-dataset run and a mid-run eviction genuinely hurts.
Storage costs: Network volumes (shared across nodes) and container disk are both billed per GB per month, roughly $0.07/GB for network storage under 1 TB ($0.05 above 1 TB) and $0.10/GB for container disk. Size the volume to your checkpoint and dataset footprint. Around 200 GB per node is a reasonable starting point, more if you are working with a 70B model and saving full optimizer states at each checkpoint. Even a few hundred GB of shared network volume runs a few dollars a month, negligible next to GPU spend.
Auto-shutdown is the cost lever that matters most. Idle GPUs keep billing, so configure an idle GPU utilization threshold, a sustained drop below a few percent is a reliable signal your job has crashed, and you stop paying for H100s that are doing nothing while an exception sits silently in a log file. The setting lives in cluster configuration at provisioning time.
Which tier to choose:
- Secure Cloud for production training with hard deadlines or compliance requirements
- Community Cloud for exploratory runs, hyperparameter sweeps, or fine-tuning jobs where you have built restart logic into your training loop
- Serverless does not fit multi-node training
Provisioning a Multi-Node H100 Cluster via Runpod Instant Clusters
The provisioning itself is fast, well under a minute. The time sink is landing in the right part of the console, which is not where most teams look first.
Get the navigation right before anything else. The Clusters section is separate from the Cloud tab. The Cloud tab covers single-pod deployments; Instant Clusters has its own section in the console navigation. If you are on the Cloud tab trying to wire up multi-node training, you are in the wrong place.
Step 1: Create the cluster
From the Clusters page, click Create Cluster. Configure:
- Cluster name: something you will recognize in pod logs
- GPU type: H100 SXM for this walkthrough
- Pod Count: the number of nodes (2–8; each pod is one node)
- GPUs per node: up to 8 per node
For a 2-node × 8×H100 SXM cluster, set Pod Count to 2 and 8 GPUs per node. Runpod shows the estimated hourly rate before you confirm. Click Deploy Cluster. Once provisioning completes, you land back on the Clusters page with your new cluster visible.
Step 2: Docker image selection
Docker image selection is the step most guides skip, and a large share of multi-node problems trace back to it. Use Runpod's recommended base image, the "Runpod PyTorch" template, built around Python 3.10+, PyTorch 2.x, and CUDA 12, with DeepSpeed, Fully Sharded Data Parallel (FSDP), Hugging Face Transformers, and the NVIDIA Collective Communications Library (NCCL) available.
The reason is version coherence: NCCL version mismatches between nodes silently hang your all-reduce operations, and the Runpod base is designed to keep NCCL consistent across every pod in the cluster.
Bring a custom image only if your dependencies genuinely conflict with the base. If you do, pin NCCL and PyTorch versions in your Dockerfile and test them on a single-node pod before spinning up a cluster.
Step 3: Storage configuration
Attach a network volume for model checkpoints and datasets. This gives every node in the cluster shared storage. For a 2-node run with a 7B–70B model, a few hundred GB is a reasonable baseline. Add a per-node container volume for ephemeral working data, like tokenized batches each node preprocesses independently.
High-speed node connections are active by default on SXM clusters, up to 3200 Gbps node-to-node on H100. The RDMA fabric (InfiniBand or RoCE v2, depending on the region) comes up automatically; you do not configure it here, but you will point NCCL at it in Section 5.
Step 4: Auto-shutdown and access setup
Set your idle GPU utilization threshold before deploying. In cluster settings, configure the cluster to shut down after GPU utilization stays below a low threshold for a sustained window. On a long training run, that is the cheapest insurance available.
Once the cluster is live, you will see individual pods listed under it: CLUSTERNAME-pod-0, CLUSTERNAME-pod-1, and so on. Connect to each via SSH or the web terminal. For SSH, generate an ed25519 key locally:
ssh-keygen -t ed25519 -C "your@email.com"Add the public key in Runpod Account Settings under SSH Public Keys. Runpod injects it into ~/.ssh/authorized_keys when pods start. Use the SSH command from each pod's Connect dialog.
Step 5: Secrets management
Keep HF_TOKEN, W&B (Weights & Biases) API keys, and any other credentials out of your container config and training-script environment. Use Runpod Secrets Management instead. Navigate to the Secrets section in the console, click Create Secret, and set a name like huggingface_token with your token as the value.
When creating the cluster template, reference the secret as an environment variable with the syntax {{ RUNPOD_SECRET_huggingface_token }}. At runtime it resolves to the secret value inside the container as a standard environment variable, reachable via os.environ.get('huggingface_token') in your training script, without the value ever appearing in container config.
Configuring and Launching a Distributed Training Job
A distributed job needs three things settled before you commit compute: an identical environment on every node, a correct NCCL interface, and a rendezvous you have actually tested.
Once the cluster is live and you have SSH access to both pods, the per-node setup is the same. Clone your training repo, install dependencies, and verify NCCL is reachable:
pip install -r requirements.txt
python -c "import torch; print(torch.cuda.nccl.version())"You should see an NCCL version tuple. An error here points at your base image, so sort it before you try to launch across nodes.
Get the primary node's internal IP from the Runpod pod dashboard, listed under the pod's network details. That address is your MASTER_ADDR. Export it on both nodes so the launch commands can reference it:
export MASTER_ADDR=<pod-0-internal-ip>Cluster topology
A 2-node × 8×H100 SXM cluster on Runpod connects two pods over a high-speed RDMA fabric (InfiniBand or RoCE v2, depending on the region), with each pod's eight GPUs communicating internally at 900 GB/s over NVLink:
Between nodes, traffic runs over the RDMA fabric on interfaces ens1 through ens8, up to 3200 Gbps aggregate on H100. Runpod exposes these eight high-bandwidth interfaces per pod for inter-node traffic, separate from eth0, the external internet interface.
NCCL environment variables
Most multi-node setups on Runpod break at NCCL configuration. You have to point NCCL at the cluster's internal high-speed interfaces. Without that, all-reduce traffic routes over the external eth0 interface (the 172.x range reserved for internet), and you get connection timeouts or throughput that looks like your GPUs are talking through a VPN.
The one setting Runpod documents as required is the interface binding, plus debug logging for the first run:
export NCCL_SOCKET_IFNAME=ens1 # Bind NCCL to the internal cluster fabric, not eth0
export NCCL_DEBUG=INFO # Verbose logging; useful for diagnosing first-run hangsNCCL_SOCKET_IFNAME=ens1 is the load-bearing line: it pins NCCL to the high-bandwidth internal network instead of letting it fall back to eth0. To spread traffic across more of the fabric, pass a comma-separated list (ens1,ens2,ens3,...).
If your cluster exposes RDMA verbs devices (InfiniBand or RoCE), you can additionally turn on the RDMA path for a further latency cut on large tensors:
export NCCL_IB_DISABLE=0 # Use the RDMA (IB/RoCE) transport when verbs devices are present
export NCCL_NET_GDR_LEVEL=2 # GPUDirect RDMA: move data straight between NIC and GPU memoryThese two are advanced tuning, not part of Runpod's documented baseline, and they help only when the verbs devices are actually exposed to the container. NCCL_IB_DISABLE=0 enables NCCL's RDMA transport, which covers both InfiniBand and RoCE; NCCL_NET_GDR_LEVEL=2 turns on GPUDirect RDMA so gradient tensors move directly between the NIC and GPU memory, and the benefit grows with tensor size. Where the RDMA path is not exposed, NCCL falls back to the socket transport over ens1, which is what Runpod's own quickstart uses. Set whichever apply in Runpod's pod configuration so they are present at container start, or export them at the top of your launch script. If a job hangs at startup, check the interface binding first, then confirm the fabric with ibstat before second-guessing the rest.
Launching with DeepSpeed
For multi-node DeepSpeed, launch on each node with its own --node_rank. On the primary node (rank 0):
deepspeed --num_nodes 2 \
--num_gpus 8 \
--node_rank 0 \
--master_addr $MASTER_ADDR \
--master_port 29500 \
train.py \
--deepspeed ds_config.jsonRun the same command on the secondary node with --node_rank 1. Both nodes rendezvous through MASTER_ADDR:29500, and DeepSpeed handles the rest. (If you would rather launch once from the primary, DeepSpeed can pdsh into the workers instead, given a --hostfile and passwordless SSH between pods.)
An annotated ds_config.json for a ZeRO-3 run with bf16 and NVMe offloading follows. The // lines are annotations for readability; strip them before you use the file, since DeepSpeed expects strict JSON.
{
"train_batch_size": "auto",
"train_micro_batch_size_per_gpu": "auto",
"bf16": {
"enabled": true // bf16 preferred over fp16 on H100; avoids loss scaling
},
"zero_optimization": {
"stage": 3, // ZeRO-3: partitions weights + gradients + optimizer states
"allgather_partitions": true,
"reduce_scatter": true,
"allgather_bucket_size": 2e8,
"reduce_bucket_size": 2e8,
"overlap_comm": true, // Overlap communication with compute
"contiguous_gradients": true,
"offload_optimizer": {
"device": "nvme",
"nvme_path": "/workspace", // Runpod NVMe-backed storage mount point
"pin_memory": true,
"buffer_count": 4,
"fast_init": false
},
"offload_param": {
"device": "nvme",
"nvme_path": "/workspace",
"pin_memory": true,
"buffer_count": 5,
"buffer_size": 1e8
}
},
"optimizer": {
"type": "AdamW",
"params": {
"lr": 1e-4,
"betas": [0.9, 0.95],
"eps": 1e-8,
"weight_decay": 0.01
}
},
"gradient_clipping": 1.0,
"steps_per_print": 100,
"wall_clock_breakdown": false
}
allgather_bucket_size and reduce_bucket_size at 2e8 (200 million elements) are common starting points; on an H100 cluster with a fast RDMA fabric, tune upward toward 5e8 if communication is not fully overlapping with compute. Keep overlap_comm: true on by default, since dropping it leaves throughput on the table (the exception is some CPU-offload configurations, where full overlap is harder to reach). The offload_optimizer and offload_param sections use nvme_path: /workspace, the default Runpod container volume mount. Confirm your pod's disk is set to NVMe, and it will absorb the checkpoint I/O without becoming a bottleneck for models up into the tens of billions of parameters.
Launching with torchrun
Launch native PyTorch FSDP with torchrun on each node. Hugging Face Accelerate wraps the same distributed launch, so if you prefer it, generate a multi-node config with accelerate config and swap torchrun for accelerate launch --multi_gpu.
On Node 0 (primary):
torchrun \
--nproc_per_node=8 \
--nnodes=2 \
--node_rank=0 \
--master_addr=$MASTER_ADDR \
--master_port=29500 \
train.pyOn Node 1 (secondary):
torchrun \
--nproc_per_node=8 \
--nnodes=2 \
--node_rank=1 \
--master_addr=$MASTER_ADDR \
--master_port=29500 \
train.pyMASTER_ADDR is the internal IP of pod-0, visible in the pod's network details in the Runpod dashboard. Port 29500 is the conventional torchrun rendezvous port; pick one that is free and use it consistently across both nodes. In your training script, bind each process to its local GPU before initializing the process group, otherwise every rank on a node defaults to GPU 0:
import os
import torch
import torch.distributed as dist
local_rank = int(os.environ["LOCAL_RANK"])
torch.cuda.set_device(local_rank)
dist.init_process_group(backend="nccl")
NCCL picks up the NCCL_SOCKET_IFNAME and any RDMA settings you exported earlier automatically.
First-run validation
Before committing to a full training run, validate multi-node connectivity with a minimal script that runs a distributed all-reduce across both nodes:
import torch
import torch.distributed as dist
dist.init_process_group(backend="nccl")
rank = dist.get_rank()
world_size = dist.get_world_size()
tensor = torch.ones(1).cuda(rank % 8) # One value per process
dist.all_reduce(tensor, op=dist.ReduceOp.SUM)
if rank == 0:
assert tensor.item() == world_size, f"Expected {world_size}, got {tensor.item()}"
print(f"All-reduce validated across {world_size} processes")If this completes without hanging, your NCCL configuration is working and both nodes are talking over the cluster's high-speed interconnect (InfiniBand or RoCE v2, depending on configuration).
Optimization and What to Expect from Performance
Whether a Runpod H100 cluster scales well comes down almost entirely to NCCL topology. The settings below are what separate a cluster that tracks its GPU count from one that stalls on communication.
ZeRO-3 for models above 6B parameters
ZeRO-3 partitions weights, gradients, and optimizer states across every GPU in the cluster, which is what makes 70B+ models trainable on a 2-node × 8×H100 setup without pipeline parallelism. The memory math explains why it stops being optional at this scale.
Runpod's documented recommendation is ZeRO-3 for models above 6B parameters. A 7B parameter model in fp16 needs on the order of 84 GB per GPU in a naive data-parallel setup: roughly 14 GB for weights (7B × 2 bytes), another ~14 GB for gradients, and ~56 GB for Adam optimizer states in fp32. That already exceeds a single H100's 80 GB VRAM.
Partition all three components across GPUs and the picture changes. On a 2-node × 8×H100 cluster (16 GPUs total), a 7B model's weight tensor splits across all 16 processes, dropping per-GPU weight memory to roughly 875 MB, effectively negligible. You recover the full 80 GB per GPU for activations, intermediate tensors, and batch data, all without the implementation complexity and lower hardware utilization that pipeline parallelism would otherwise add.
ZeRO-3's communication cost is the all-gather before the forward pass and the reduce-scatter after the backward pass. On a cluster with a fast RDMA fabric, and GPUDirect RDMA enabled where available, that overhead stays manageable, especially with overlap_comm: true in your ds_config, which pipelines communication against compute so your GPUs are not idling on the all-gather.
NVMe offloading for large models on PCIe nodes
NVMe offloading extends the trainable model size on PCIe nodes beyond what VRAM alone allows, routing optimizer states and parameters to Runpod's NVMe-backed storage between the forward and backward passes. That per-step round trip is a different operation from checkpoint I/O, and it does add latency, which is the trade-off you are accepting.
If you are on PCIe nodes for cost reasons and training a larger model, offloading becomes the practical way to fit it. The offload_optimizer and offload_param settings in the ds_config above handle it automatically. A model that would OOM on an 8×H100 PCIe node under ZeRO-2 can become trainable with ZeRO-3 plus NVMe offload. You pay for it in throughput, since NVMe I/O adds latency on the offload/load cycle. For exploratory runs on a tight budget, that is often a trade worth making.
FP8 training
FP8 training halves the memory footprint of activations and weights versus fp16/bf16 and lifts throughput on transformer architectures. It is available on all three H100 variants via NVIDIA's Transformer Engine, which Runpod's training environments support.
FP8 is not a DeepSpeed config flag. You enable it in the training loop through Transformer Engine, and it composes with ZeRO stages 0 through 3. Build your model from Transformer Engine modules (te.Linear, te.LayerNorm, and so on), then wrap the forward pass in an fp8_autocast context:
import transformer_engine.pytorch as te
from transformer_engine.common import recipe
# E4M3 on the forward pass, E5M2 on the backward pass
fp8_recipe = recipe.DelayedScaling(fp8_format=recipe.Format.HYBRID,
amax_history_len=16,
amax_compute_algo="max")
model, optimizer, _, _ = deepspeed.initialize(model=model, config=ds_config)
for batch in batches:
with te.fp8_autocast(enabled=True, fp8_recipe=fp8_recipe):
output = model(batch)
loss = output.mean()
model.backward(loss)
model.step()Only the forward pass goes inside the context; DeepSpeed handles the backward and step as usual, and the recipe's defaults are sensible, so amax_history_len and amax_compute_algo are optional knobs. On the Hugging Face Accelerate path you get the same result by setting mixed_precision: fp8 in your Accelerate config, which uses Transformer Engine as the backend. Either way the layers must be Transformer Engine modules; a stock nn.Linear sees no FP8 benefit. LLaMA-3 fine-tuning with Transformer Engine is well documented; Mistral and Qwen may need a bit more setup.
What to expect on throughput
NVIDIA's Hopper Transformer Engine benchmarks put H100 SXM at up to about 4× the training throughput of A100 SXM on transformer workloads, driven mainly by its 3.35 TB/s HBM3 bandwidth, faster NVLink, and FP8 acceleration. That is the per-GPU uplift.
Actual tokens-per-second for a LLaMA-3-8B fine-tune with DeepSpeed ZeRO-3 and bf16 depends heavily on your dataset and batch configuration, so treat any single published figure with suspicion and benchmark your own setup. As a rough scaling expectation, a well-configured second node with good NCCL and fabric scaling should come close to doubling single-node throughput, minus per-step communication overhead.
The all-reduce validation script in Section 5 is your starting point for that; extend it to profile a few training steps with torch.profiler.profile before you commit to a long run.
Configuration errors, a wrong NCCL interface, mismatched NCCL versions, or traffic falling back to eth0, can leave a larger H100 cluster performing worse than a smaller, well-configured one. That is why the topology details in Sections 4 and 5 earn their space.
H100 clusters on Runpod FAQ
How do I choose between H100 PCIe and H100 SXM5 on Runpod for multi-node training? Choose H100 SXM5 for any multi-node job that uses tensor parallelism above degree 2, pipeline parallelism, or pre-training at scale. SXM5 delivers 900 GB/s bidirectional NVLink bandwidth within a node, against roughly 64 GB/s over the PCIe bus on the PCIe variant, and that gap sets how far you can push tensor-parallel degree before communication dominates compute. PCIe nodes work well for cost-sensitive fine-tuning that uses pure data parallelism with ZeRO-2 or ZeRO-3, where intra-node bandwidth is not the critical path.
What causes silent hangs in NCCL on Runpod multi-node clusters? Most often, NCCL has picked the external interface instead of the internal cluster network, so set NCCL_SOCKET_IFNAME=ens1 to force the right one. The other usual suspect is a NCCL version mismatch between nodes, which happens with a custom Docker image that does not pin NCCL, and it produces a harder-to-diagnose hang, so check version consistency before you launch. If the hang comes right at startup, a quick ibstat tells you whether the fabric itself is even up before you go flag-hunting.
Why does ZeRO-3 matter specifically for models above 6B parameters on H100? Because a 7B model overruns one H100's 80 GB before training even starts. In plain fp16 data-parallel, its weights, gradients, and Adam optimizer states together want on the order of 84 GB per GPU, and the weights are the small part of that. ZeRO-3 shards all three across the cluster, so on 16 GPUs each card holds well under a gigabyte of weights and gets its full 80 GB back for activations and batch data, with no pipeline parallelism needed. Below roughly 6B the numbers still fit in one card, so the sharding overhead is not worth paying.
How does GPUDirect RDMA improve training throughput on Runpod H100 clusters? It removes a copy. Normally a gradient tensor is staged through host CPU memory on its way between the GPU and the network card; with GPUDirect RDMA (NCCL_NET_GDR_LEVEL=2, where the cluster exposes the RDMA path) it travels straight between NIC and GPU. The larger the tensor, the more that saved hop shows up as lower all-reduce latency. Pair it with overlap_comm: true and the collectives tuck behind compute instead of stalling the GPUs.
What is the safest way to store API credentials in a Runpod cluster? Use Runpod Secrets Management rather than embedding credentials in container config or your training-script environment. Create a named secret in the console and reference it in your cluster template with the syntax {{ RUNPOD_SECRET_your_secret_name }}. At runtime it resolves to the value as a standard environment variable inside the container, reachable via os.environ.get(), without the credential ever landing in container configuration or logs.
How should I validate a Runpod multi-node cluster before committing to a long training run? Run a minimal all-reduce script that starts the NCCL process group across both nodes, does one distributed all-reduce, and asserts the result equals world size. If it finishes instead of hanging, your NCCL and interconnect setup is sound. Then run a 100-step loop with your real model and batch size and torch.profiler on, to read GPU utilization and per-step timing, all for a few dollars of compute.
What auto-shutdown threshold should I configure for H100 cluster training jobs? Configure the cluster to shut down after GPU utilization stays below a low threshold for a sustained window, low enough that a dropped or crashed process trips it reliably without false positives during normal training. The setting lives in cluster configuration at provisioning time. On a 16×H100 cluster at roughly $2.99/GPU/hr, a crashed job sitting idle for four hours costs nearly $200 for no output, which makes this the highest-return configuration step for the time it takes.
Launch a Cluster in the Next 10 Minutes
You have the variant, the pricing model, the NCCL settings, and the launch commands. The last step is to put them on real hardware before you commit a long run to it.
Go to runpod.io/gpu-models/h100-sxm, click Deploy, and navigate to Instant Clusters. Create a 2-node × 8×H100 SXM cluster using the Runpod PyTorch 2.x / CUDA 12 / DeepSpeed base image. Attach a network volume sized to your model and dataset. Set auto-shutdown on a low idle-utilization threshold.
Before you run a full training job, spend a few minutes validating the setup:
- Run the all-reduce validation script from Section 5 to confirm multi-node connectivity
- Run a 100-step training loop with your actual model and batch configuration, torch.profiler active, to read GPU utilization and per-step timing
That test cycle, provisioning through validation, costs little on per-second billing, well worth it before you commit thousands of dollars of compute to a full run. Runpod's Instant Clusters documentation carries reference config templates and the full environment-variable reference if you need to dig deeper.
