DeepSeek V3 is a 671B-parameter Mixture-of-Experts model with 256 experts and only 37B parameters activated per token. That efficiency is what makes it usable at all, but it doesn't make fine-tuning cheap: even in FP8, the base model alone occupies roughly 671 GB of GPU memory, more than a single 8xH100 node's 640 GB. This guide covers what fine-tuning DeepSeek V3 actually requires and how to run it on Runpod Clusters.
1. Why this isn't a single-pod job
Full fine-tuning of DeepSeek V3 is out of reach for nearly everyone; the compute and memory requirements scale with the same 671B parameter count used in pretraining. LoRA fine-tuning changes the math substantially: published recipes bring DeepSeek V3/R1 671B LoRA fine-tuning down to roughly 24x H100 GPUs using mixed parallelism (ZeRO, pipeline parallel, expert parallel, and CPU offloading), which is still a serious cluster but roughly a tenth of naive full-parameter training hardware.
If your actual goal is a smaller, faster model with DeepSeek-style reasoning behavior, consider whether a distilled DeepSeek variant fits your task before committing to 671B-scale fine-tuning; several distilled checkpoints in the 7B-70B range are fine-tunable on a single Runpod Pod using the same QLoRA workflow covered in the best GPU for fine-tuning guide.
2. Hardware for full-scale DeepSeek V3 LoRA fine-tuning
Because the model doesn't fit on one node, this is a Clusters job from the start, not a single Pod you scale up later.
- ~24x H100 SXM across multiple nodes, using a framework with expert parallelism and CPU offloading, is the published floor for LoRA fine-tuning at 671B scale
- Multi-node InfiniBand networking matters here more than in dense-model training, since 256-expert MoE layers generate substantial all-to-all communication between GPUs during the expert routing step
There's also a numerics detail specific to DeepSeek V3: it was natively trained in FP8. For nearly all existing LoRA methods, adapter weights are assumed to run in bfloat16, and merging a bfloat16-trained LoRA into an FP8-served base model introduces a real accuracy penalty. Quantization-aware training (QAT) approaches address this by simulating FP8 numerics during the forward pass while keeping LoRA gradients in bfloat16, which is worth investigating if you plan to serve the merged model at FP8 rather than bfloat16.
3. Framework choice
Two paths handle DeepSeek V3's scale and MoE structure:
- NeMo (NVIDIA's framework) handles FP8 and MoE parallelism natively and is built for multi-node training, at the cost of converting Hugging Face weights to NeMo format first
- Colossal-AI ships a purpose-built LoRA fine-tuning recipe for DeepSeek V3/R1 671B, including a one-click script and support for ZeRO, pipeline parallel, expert parallel, and CPU offloading in combination
Colossal-AI's recipe is the more direct path if you're starting from Hugging Face weights and want a documented multi-GPU LoRA setup without a separate format-conversion step.
4. Setting up Clusters
Provision Runpod Clusters with H100 SXM nodes on the same Node Group so InfiniBand networking is available between them. Node count depends on the parallelism strategy you choose; the ~24-GPU floor above assumes a mixed-parallelism recipe with CPU offloading, and needing fewer or more GPUs depends on your batch size, sequence length, and how aggressively you offload optimizer state to host RAM.
Attach Network Volume storage sized for the 671B checkpoint (hundreds of GB even at FP8) plus your dataset and training checkpoints, since re-downloading the base model on every cluster restart is not something you want to repeat.
5. Validate before you scale
Test your training configuration, dataset formatting, and framework setup on a much smaller model first, ideally a distilled DeepSeek checkpoint on a single GPU. A parallelism misconfiguration that costs a few dollars to discover on one GPU costs dramatically more when it surfaces three hours into a 24-GPU run.
6. From adapter to serving
Once the LoRA adapter is trained, test the merged (or adapter-attached) model with an inference backend that supports DeepSeek V3's MoE architecture: vLLM, SGLang, and LMDeploy all support it. Serving at scale still requires multi-node inference (DeepSeek V3 doesn't fit on a single 80GB-per-GPU node either), so plan the serving cluster size alongside the training cluster rather than as an afterthought.
7. Quick-start checklist
- Confirm you actually need 671B-scale fine-tuning rather than a distilled variant
- Provision Runpod Clusters with InfiniBand-connected H100 SXM nodes, roughly 24 GPUs as a starting point for LoRA
- Choose Colossal-AI's DeepSeek LoRA recipe or NeMo depending on your starting weight format
- Validate the training config on a smaller model on one GPU before scaling out
- Plan the serving cluster size for the fine-tuned model alongside the training cluster
