Make the model yours
Customizability is the most underrated idea in AI right now. Runpod CEO Zhen Lu on why a model tuned on your data beats a bigger one on the job you actually have.
All
Full-parameter fine-tuning needs about eight times the memory a model's weights occupy, which is why the usual sizing advice is off by close to an order of magnitude. The rule of thumb most people work from is about 2 GB of VRAM per billion parameters. That is correct for inference, where you only hold weights to serve a model. Training has a different equation.
A 7B model in BF16 is about 14 GB of weights, so an 80 GB card looks generous. Full-parameter fine-tuning that same model with AdamW needs roughly 112 GB of resident state before a single activation tensor is allocated. That gap is where runs die with a CUDA out-of-memory error in the first few hundred steps.
This post is for the case where you have already ruled out adapters. If LoRA gives you the domain shift you need, use it: it trains a few million parameters instead of eight billion, and an 8B model fits on a 24 GB card. Full-parameter training is what you reach for when LoRA is not enough, and it is where sizing goes wrong.
Our Forward Deployed Engineering team answers this often enough to have written the size calculations down internally. What follows is that calculation, generalized into four equations you can run on any model, at any sequence length, on any GPU count. The parameter equation is checked against published counts below. Everything downstream of it is arithmetic rather than measured peak memory, so treat the totals as sizing estimates to confirm against your own allocator, not as benchmarks.
The short version for a 7B model: 112 GB of resident state, which no single 80 GB card holds. Two A100 80GB cards running DeepSpeed ZeRO stage 1 is the cheapest configuration that fits, at roughly 10 to 12 dollars of compute for a 50M-token run on an 8B model. A single H200 fits the same job on one card at a higher cost per run.
Note that all memory figures below are decimal gigabytes, where 1 GB is 10^9 bytes. GPU nameplate capacity is not: an "80 GB" A100 is 80 GiB, or 85.9 GB decimal. Mixing the conventions is how people talk themselves into and out of configurations that actually fit, so convert both sides before comparing.
Step one: compute your actual parameter count
Every number downstream is a multiple of your parameter count, and most people take that count from the model's name, but the name is marketing rounding.
For a modern decoder-only transformer with grouped-query attention and a SwiGLU feed-forward block:
P = L · [ 2·D_h² + 2·D_h²·(n_kv/n_h) + 3·D_h·D_ff ] + V·D_h·(2 if untied embeddings else 1)
L is layer count, D_h is hidden size, n_h is attention heads, n_kv is key/value heads, D_ff is the feed-forward inner dimension, and V is vocabulary size. All of these sit in the model's config.json.
The terms map onto the architecture directly: 2·D_h² is the query and output projections, 2·D_h²·(n_kv/n_h) is the key and value projections shrunk by the grouped-query ratio, and 3·D_h·D_ff is the SwiGLU gate, up, and down projections. With classic multi-head attention n_kv/n_h is 1 and the attention term collapses to 4·D_h².
Against models where the answer is published:
Three models the industry calls 7B or 8B span 6.74 to 8.03 billion parameters, a 19 percent spread. At 16 bytes per parameter that is 108 GB against 128 GB, which decides whether a configuration fits on two cards. Compute P. Do not look it up.
Step two: the five tensors resident in VRAM
Training holds five distinct tensors per parameter:
M_state = P · (b_w + b_g + b_m + b_v1 + b_v2)
Weights used in the forward and backward pass are BF16, so 2 bytes. Gradients are one per trainable parameter, usually BF16, so 2 bytes. The FP32 master copy is 4 bytes. Adam's first and second moments are FP32, 4 bytes each. That sums to 16 bytes per parameter, or 112 GB for a 7B model. The 14 GB you started with is the smallest of the five terms.
The constant depends on your recipe, and the spread is wide:
The 8-bit rows mean bitsandbytes, exposed in most trainers as optim="adamw_bnb_8bit", which quantizes the moment buffers block-wise. It is well behaved on standard runs, but it is a different optimizer, so validate on a short one first. To measure what your framework actually does rather than assuming 16 bytes, take one optimizer step and check both halves:
weights = sum(p.numel() * p.element_size() for p in model.parameters())opt = sum(t.numel() * t.element_size() for s in optimizer.state.values() for t in s.values() if torch.is_tensor(t))print(weights / 1e9, opt / 1e9, (weights + opt) / sum(p.numel() for p in model.parameters()))The third figure is your real bytes per parameter, master copy and moments included.
What Adam is actually storing
Plain SGD needs no state: apply w ← w − η·g and discard the gradient. Adam maintains two running statistics per parameter:
m ← β₁·m + (1−β₁)·g first moment, smoothed gradient direction
v ← β₂·v + (1−β₂)·g² second moment, smoothed gradient magnitude
w ← w − η · m̂ / (√v̂ + ε) the update, scaled per parameter
The first moment is momentum: recent gradients of +0.10, +0.12, +0.09, and +0.11 give a value near +0.105, so one anomalous step does not yank the direction around. The second moment averages squared gradients, which discards sign and exposes volatility, so Adam takes smaller steps where updates are erratic. Both are FP32, 4 bytes each, together 56 GB for a 7B model.
People often quote Adam's optimizer state as 84 GB for a 7B model, or 12 bytes per parameter. The two moments alone are 56 GB, or 8 bytes. The remaining 28 GB is the FP32 master copy, which mixed precision requires and Adam does not. An 8-bit optimizer compresses the moments from 8 bytes to 2 and leaves the master copy alone, so budgeting 12 bytes for the optimizer means expecting a saving that does not arrive.
The master copy stops small updates vanishing. BF16 gives 8 exponent bits and 7 mantissa bits, so near 1.0 the spacing between representable values is about 0.0078. An update of 0.00001 is three orders of magnitude below that gap and rounds away to nothing. Thousands of lost updates leave the model stalled or unstable, and the symptom looks like a bad learning rate.
Step three: activations, and why FlashAttention is not optional
Activations sit on top of resident state. During inference a layer's output is discarded once the next layer consumes it. During training the backward pass needs those intermediates, so they stay resident, which is why training costs more than serving.
The standard reference is Korthikanti et al., Reducing Activation Recomputation in Large Transformer Models. Per transformer layer:
A_layer = S · B · D_h · (34 + 5·n_h·S/D_h) bytes
S is sequence length and B is micro-batch size. Total activation memory is L · A_layer. The 34·S·B·D_h term is linear in sequence length. The 5·n_h·S²·B term is the materialized attention score matrix, quadratic in sequence length, and FlashAttention eliminates it by never materializing the full matrix.
The constant 34 was derived for a GPT-3-style architecture with multi-head attention and a two-matrix feed-forward block. On a model with grouped-query attention and SwiGLU it is an approximation, so treat the figures below as accurate to roughly 20 percent and instrument torch.cuda.max_memory_allocated() for the real number.
Using Llama 3 8B geometry at micro-batch 1:
At 4K context FlashAttention removes about 86 GB, more than the model state. It also makes activation memory linear in sequence length, so doubling context doubles activations rather than quadrupling them. The familiar warning about quadratic attention memory is obsolete for anyone using a modern kernel and it leads teams to over-provision.
The FlashAttention backward pass is non-deterministic, so bit-exact reproducibility across runs goes away, and custom attention masks or additive biases are limited to what the kernel supports. For almost every fine-tuning job those are acceptable prices for 86 GB.
Gradient checkpointing: what recomputation costs
Full activation checkpointing stores only each layer's input, 2·S·B·D_h, the last column of that table. Training compute is approximately C ≈ 6·P·D for P parameters and D tokens, split 2PD forward and 4PD backward. Checkpointing reruns the forward pass during the backward pass, so C ≈ 8·P·D, about 33 percent more. Selective recomputation, which checkpoints only the attention block, costs 5 to 10 percent.
DeepSpeed ZeRO stages, and why stage 3 is usually the wrong default
When the model state does not fit, you shard it. For N GPUs, with weights at 2P, gradients at 2P, and optimizer state at 12P:
DDP M/GPU = 2P + 2P + 12P = 16P
ZeRO-1 M/GPU = 2P + 2P + 12P/N
ZeRO-2 M/GPU = 2P + (2P + 12P)/N
ZeRO-3 M/GPU = (2P + 2P + 12P)/N = 16P/N
PyTorch FSDP full-shard is equivalent to ZeRO-3. For a 7B model, in GB per GPU:
Memory is half the choice. Per step, DDP, ZeRO-1, and ZeRO-2 all move roughly 2P of gradient traffic. ZeRO-3 also gathers parameters in the forward and backward pass, bringing it to 3P.
That 1.5x is traffic volume, and how much of it reaches your step time depends on the regime. Inside one node on NVLinked SXM cards with sensible bucketing, the extra gathers overlap with compute and the penalty is often small. Across nodes on Ethernet it can exceed 1.5x, because there is no longer motherboard bus level bandwidth to hide behind. Pick the lowest stage that fits and measure the rest. A 7B model on two 80 GB cards fits at ZeRO-1, which makes ZeRO-3's gathers a cost with nothing to show for it.
Sharding happens at launch, not in the config file
Adding an FSDP or DeepSpeed block to a config file shards nothing on its own. Without a distributed launcher you get ValueError: Using fsdp only works in distributed training on a machine where all eight GPUs are visible and idle.
A minimal stage 1 config, ds_zero1.json. The "auto" values let the launcher fill them from your training arguments, avoiding the error you get when JSON and command line disagree:
{
"bf16": { "enabled": "auto" },
"zero_optimization": {
"stage": 1
},
"gradient_clipping": "auto",
"train_micro_batch_size_per_gpu": "auto",
"gradient_accumulation_steps": "auto"
}
Launched on a two-GPU pod, assuming train.py uses the Hugging Face Trainer. With a custom training loop you would call deepspeed.initialize() instead and pass the same config:
export PYTORCH\_CUDA\_ALLOC\_CONF=expandable\_segments:True
deepspeed \--num\_gpus 2 train.py \\
\--deepspeed ds\_zero1.json \\
\--bf16 \--gradient\_checkpointing \\
\--per\_device\_train\_batch\_size 1 \\
\--gradient\_accumulation\_steps 16expandable_segments:True reduces allocator fragmentation, which is why jobs that should fit still fail. An out-of-memory error triggers on peak reserved rather than peak allocated, and the gap can run to several gigabytes. Track both with torch.cuda.max_memory_reserved() and torch.cuda.max_memory_allocated().
Every figure here assumes micro-batch 1, the smallest activation footprint. Effective batch size is micro-batch times accumulation steps times GPU count, so 1 by 16 by 2 gives 32 sequences per step. Accumulation costs no memory, so raise it until your effective batch is right and leave micro-batch alone.
Putting it together
M_total/GPU = M_state(N, stage) + L·A_layer(S, B, D_h, n_h) + M_overhead
M_overhead covers the CUDA context, NCCL buffers, cuBLAS and cuDNN workspaces, and fragmentation. Budget 2 to 4 GB per GPU, because fragmentation runs higher than expected and reserved memory is what ends the run.
Llama 3 8B, sequence 4,096, micro-batch 1, BF16 AdamW, FlashAttention and full checkpointing on, 3 GB overhead:
ZeRO-1 fits, but a 1.6 GB margin on an 84 GB working set sits inside the error bars of both the activation approximation and the overhead estimate. Try ZeRO-1, keep ZeRO-2 as the fallback, and confirm on your own hardware rather than on this table.
A100 vs H100 vs H200 for a full fine-tune
The standard advice was written when 80 GB was the ceiling. Against 112 GB of model state for a 7B model plus about 4 GB of activations and overhead:
Those are vendor nameplate figures, and they behave as GiB rather than decimal gigabytes: nvidia-smi on an A100 80GB reports 81,920 MiB, exactly 80 GiB, or 85.9 GB decimal, and the same holds across the range. Usable capacity is lower once the CUDA context is allocated, so check nvidia-smi on the card you rented rather than the label or this table.
The MI300X capacity is real, but every other card there runs CUDA and this post assumes a CUDA stack. The AMD path is ROCm, where framework support and images differ materially, so this advice does not transfer to a row picked purely on capacity.
Hourly rates do not tell you what a job costs. Convert FLOPs to hours with T = C / (N · F_peak · MFU), where F_peak is your card's dense BF16 throughput and MFU is model FLOPs utilization. Sharding costs MFU, because gradient reduction and parameter gathers never overlap with compute perfectly, so model a sharded configuration lower than a single card doing the same work. Using C = 8·P·D for a 50M-token run on Llama 3 8B, at 0.35 to 0.42 sharded and 0.42 to 0.50 single-card:
Two A100s is the cheapest configuration that fits, and that ordering survives the MFU assumption: the pessimistic end of its band still beats the H200 at the optimistic end of its. The run-time ordering does not survive, because those ranges overlap, so an H200 with good utilization can finish ahead of an A100 pair losing more than expected to communication. Treat the A100 pair as the cheap option, not the fast one. The H100 pair is the fast one, a few dollars more per run.
The H200 is the most expensive per run and still often the right choice, for a reason the table cannot show. There is no distributed launcher, no NCCL configuration, and no sharding strategy to get wrong. If you measure the hardware cost against a day of engineering time debugging a two-GPU setup, then the ten dollars of compute may not be very important to you.
The cheaper capacity, and what it requires
Everything above is Secure Cloud. Community Cloud aggregates capacity from vetted third-party providers at 20 to 40 percent less, with dynamic pricing and no uptime guarantee, which rules it out for a customer-facing API but not for a training run that can resume. Spot Pods trade interruptibility for a lower rate again.
The failure mode on both is not the one people picture. Losing an hour of compute is the small cost. The exposure is that capacity may not be immediately available again, so a two-day run can stretch across a longer window while a deadline stays put. When the schedule binds, pay on-demand. For sustained programs, savings plans cover GPU compute on 3-month or 6-month commitments, excluding storage and non-refundable, so size any commitment against your floor rather than your peak.
Practical notes for long runs
Checkpoint every hour to every few hours. A checkpoint is not the full 16 bytes per parameter: you write the FP32 master weights and the two Adam moments, 12 bytes per parameter, or 84 GB for a 7B model. Gradients are never saved and the BF16 copy is derived from the master weights. At 300 MB/s, mid-range for standard network storage, that write costs about five minutes of stalled GPU. Keep optimizer state only in the newest checkpoint and the rest as weights-only copies at 28 GB each, which puts three rolling checkpoints plus a few milestones near 400 GB.
Store them somewhere durable. Container disk is cleared on stop and restart. A volume disk survives until the pod is deleted. A network volume survives independently and must be attached at pod creation; it cannot be attached later.
You can keep storage costs down too. A volume disk on a stopped pod bills at double the running rate, so pausing over a weekend with terabytes provisioned costs more than expected. Transfer in and out is free, so staging data or pulling weights down carries no egress charge.
Watch VRAM pressure rather than inferring it from crashes. On a pod, nvidia-smi --query-gpu=memory.used --format=csv -l 5 gives the signal directly. Instant Clusters expose the same through a preconfigured Grafana instance.
A single pod scales to eight GPUs, covering every configuration here. Instant Clusters are for jobs needing more than one node. If you shard across nodes, set NCCL_SOCKET_IFNAME=ens1, or nodes will try to reach each other over external addresses and fail with timeouts that look like a hardware fault. Debug with NCCL_DEBUG=INFO. Run the job inside tmux, which is not installed by default.
The procedure
- Compute P from config.json rather than from the model's name.
- Compute M_state = P · b, taking b from the recipe table. Default to 16.
- Compute activations as 34·L·S·B·D_h with FlashAttention on, or 2·L·S·B·D_h with full checkpointing on. Add 2 to 4 GB of overhead.
- Divide by GPU count at the lowest ZeRO stage that fits, converting both sides to the same unit before comparing against nameplate VRAM.
- Convert FLOPs to hours with T = 8·P·D / (N · F_peak · MFU) and multiply by the hourly rate.
Run that on Llama 3 8B at 4K and step four gives 84.3 GB per GPU against 85.9 GB of capacity. Whether 1.6 GB of headroom survives a real allocator is the one thing the equations cannot tell you.
Optimizing spend
Ultimately, successful full-parameter fine-tuning depends on rigorous estimation, not heuristics. By calculating the exact memory overhead for your specific model architecture and optimizer configuration rather than relying on generalized rules of thumb, you can circumvent costly CUDA out-of-memory errors and optimize your compute spend. As model architectures and hardware capabilities continue to evolve, mastering these underlying equations ensures you always provision exactly the resources your training jobs demand.
Get started
Runpod Serverless runs your container as an autoscaling endpoint that scales to zero and bills by the second.
- Try Serverless: Runpod console
- Python SDK: github.com/runpod/runpod-python
- Examples: github.com/runpod-workers
- Docs: docs.runpod.io/serverless/overview
- Discord: join us here
.avif)
