How to Fine-Tune GLM 5 on Runpod

GLM-5 is Zhipu AI's MIT-licensed frontier-class model, and like its GLM-4.5/4.6 predecessors it's a Mixture-of-Experts architecture, which means fine-tuning it isn't quite the same exercise as fine-tuning a dense 70B model. This guide covers the fine-tuning workflow on Runpod using Axolotl, including the MoE-specific settings that determine whether the job fits on the hardware you rent.

A note on model support: Axolotl's model support list changes with nearly every release; at the time of writing it explicitly lists GLM-4.5-Air, GLM-4.6V, and GLM-4.7-Flash. Before renting hardware for a GLM-5 job specifically, check Axolotl's current documentation for GLM-5 support and any config differences from the GLM-4.x recipes below, since exact flags and parameter names shift as support for new checkpoints lands.

1. Why MoE fine-tuning needs different settings than a dense model

A Mixture-of-Experts model routes each token to a subset of expert sub-networks rather than running it through the full parameter count. That's efficient for inference, but for training it means a naive LoRA setup would need adapter matrices for every expert in every MoE layer, and running each expert's forward pass sequentially is extremely slow. Axolotl addresses this with quantize_moe_experts: true for VRAM reduction and ScatterMoE LoRA support, which trains directly on MoE expert weights using custom Triton kernels instead of looping over experts naively.

2. GPU sizing

Treat GLM-5 like any large MoE model: total parameter count sets a ceiling, but activated parameters per token and Axolotl's MoE-specific memory optimizations determine what actually fits.

• Single A100 80GB or H100 80GB: QLoRA on GLM-5 with quantize_moe_experts: true, for models in the same weight class as GLM-4.5-Air
• Multi-GPU H100 SXM (Clusters): larger GLM-5 variants or full fine-tuning, using FSDP2 with expert parallelism

Run a smoke test on one GPU with a small data slice before committing to a larger cluster; MoE configuration mistakes are more expensive to discover after scaling out than dense-model ones.

3. Launching a Runpod Pod for fine-tuning

From Pods > GPU Cloud, deploy on an A100 80GB or H100 80GB with a PyTorch-based image, or Axolotl's published Docker image if you don't want to install the training stack yourself. Attach a Network Volume sized for the base model checkpoint plus your dataset and training checkpoints, mounted at /workspace.

Install Axolotl inside the pod:

curl -LsSf https://astral.sh/uv/install.sh | sh
export UV_TORCH_BACKEND=cu128
uv venv --python 3.12
source .venv/bin/activate
uv pip install torch==2.10.0 torchvision
uv pip install --no-build-isolation axolotl[deepspeed]
axolotl fetch examples

4. A representative Axolotl config

Start from the closest published GLM example and adjust the model identifier once GLM-5 support lands under a specific config name; the shape of the config for a LoRA job on a large MoE model looks like:

base_model: zai-org/GLM-5
load_in_4bit: true
adapter: qlora
lora_r: 16
lora_alpha: 16
lora_target_modules:
  - q_proj
  - k_proj
  - v_proj
  - o_proj
quantize_moe_experts: true

datasets:
  - path: your-dataset
    type: chat_template

sequence_len: 4096
gradient_checkpointing: true
optimizer: paged_adamw_8bit
num_epochs: 2
learning_rate: 0.0002

lora_r: 16 is a reasonable default; only increase it if validation loss plateaus early. Gradient checkpointing and a paged 8-bit optimizer are the two settings with the highest VRAM-savings-per-line-of-config on a job this size.

5. Running the job

accelerate launch -m axolotl.cli.train config.yml

Watch the first few hundred steps closely. An MoE-specific misconfiguration (wrong expert routing settings, a target-module mismatch) tends to surface as either an immediate crash or a loss curve that doesn't move, not a subtle quality regression you'd only catch at evaluation.

6. From fine-tuned adapter to serving

Once training completes, merge the LoRA adapter into the base weights or keep it separate depending on your serving setup, then test with vLLM or LMDeploy, both of which support tensor and expert parallelism for MoE models. If a single GPU handled fine-tuning but the base model needs multiple GPUs to serve, that's expected: training with QLoRA fits differently than serving the merged model at full precision.

7. Quick-start checklist

1. Confirm current GLM-5 support and config naming in Axolotl's docs before provisioning hardware
2. Start with a single A100 80GB or H100 80GB pod for QLoRA with quantize_moe_experts: true
3. Validate the recipe on a small data slice before scaling to Clusters
4. Use gradient checkpointing and a paged optimizer as the first VRAM-saving levers
5. Test the fine-tuned model with vLLM or LMDeploy before deploying

Start a fine-tuning Pod on Runpod →