Last updated: July 2026
The GPU you need for fine-tuning depends far more on method (full fine-tuning, LoRA, or QLoRA) than on model size alone. An A100 80GB or H100 80GB handles QLoRA fine-tuning on a 70B model comfortably; the same 70B model in full fine-tuning needs roughly 8x that VRAM across multiple GPUs. Get the method right first, then size the GPU.
Quick picks
- Best for most fine-tuning jobs: a single A100 80GB or H100 80GB running QLoRA. Covers 7B-70B models depending on rank and sequence length.
- Best for iterating on small models fast: RTX 4090 (24 GB) for QLoRA on 7B-13B models, where per-second billing on a cheaper card beats renting an 80GB GPU you don't need yet.
- Best for full fine-tuning or MoE models at scale: multi-node H100 SXM clusters with FSDP2 or expert parallelism, where the job genuinely needs distributed training rather than a bigger single card.
Why fine-tuning VRAM math looks nothing like inference VRAM math
Inference only needs model weights plus a KV cache. Training needs weights, gradients, optimizer states, and activations, all at once. Full fine-tuning with Adam optimizer states means a 70B model needs roughly 560 GB just for the optimizer's momentum buffers, which is why full fine-tuning of large models is a multi-GPU exercise regardless of how much VRAM a single card has.
LoRA changes the picture by freezing the base model and training only small low-rank adapter matrices, typically well under 1% of total parameters. QLoRA goes further, quantizing the frozen base model to 4-bit (NF4) while keeping the trainable adapters in bf16, which is why a 70B model that needs ~560 GB for full fine-tuning fits in roughly 46-48 GB with QLoRA.
GPU sizing by method and model size
| Model size | QLoRA | LoRA (16-bit) | Full fine-tune |
|---|---|---|---|
| 7B | ~8-10 GB (RTX 4090) | ~16-24 GB (RTX 4090/L40S) | ~60-80 GB (A100/H100) |
| 13B | ~12-16 GB (RTX 4090) | ~30-40 GB (L40S) | ~120-150 GB (2x A100/H100) |
| 70B | ~46-48 GB (A100/H100 80GB) | ~140-160 GB (2x A100/H100) | ~560+ GB (8x H100 with FSDP2) |
These are planning numbers at batch size 1, moderate sequence length, with gradient checkpointing enabled. Longer sequences and larger batches raise activation memory on top of this baseline; run a short smoke test on your actual dataset before committing to a longer job.
When you need more than one GPU
Two scenarios push a fine-tuning job from a single Runpod Pod to Clusters: the model doesn't fit on one card even with QLoRA (70B+ full fine-tuning, or any MoE model at full precision), or you want to cut wall-clock training time on a large dataset by sharding across nodes with FSDP2. Axolotl and TorchTune both support FSDP2 natively; the config change to go from single-GPU to multi-node is a few lines in the YAML, not a rewrite.
Validate your training recipe on a single GPU with a small slice of data first. A configuration mistake caught on one L40S costs a few dollars; the same mistake caught after launching an 8-node H100 cluster costs a great deal more.
Memory-saving techniques that stack
If a job is close to the VRAM ceiling, these apply in roughly this order of impact:
- Gradient checkpointing cuts activation memory 40-60% at a 25-30% compute cost. Enable this first.
- Paged AdamW (
optim="paged_adamw_32bit") offloads optimizer states to CPU RAM when VRAM gets tight, saving a few GB on 7B-32B QLoRA runs. - Lower LoRA rank. Dropping from r=64 to r=16 cuts adapter memory and optimizer state roughly 4x with minimal quality loss for most tasks. Start at r=16.
A practical fine-tuning path on Runpod
- Select the base model and fine-tuning method (start with QLoRA unless you have a specific reason not to)
- Launch a single GPU pod sized from the table above and validate the training recipe on a small data slice
- Scale to a larger single GPU, or to Clusters with FSDP2, once the recipe is confirmed working
- Test the resulting model with vLLM or TGI before committing to a production serving path
- Deploy on a Pod or Runpod Serverless depending on traffic pattern
FAQ
What GPU do I need to fine-tune a 70B model?
A single A100 80GB or H100 80GB is sufficient for QLoRA. Full fine-tuning needs roughly 8x H100 80GB with FSDP2.
Is QLoRA good enough for production, or should I use full fine-tuning?
QLoRA retains 95%+ of full fine-tuning quality on most domain-adaptation and style tasks at a fraction of the hardware cost. Reach for full fine-tuning only when you have a large, clean dataset and a specific reason LoRA-class methods haven't worked.
When do I need Clusters instead of a single Pod?
When the model and training method genuinely don't fit on one GPU's VRAM, not before. Validate the recipe on one GPU first so a configuration mistake doesn't get multiplied across a distributed job.
