Deploying Kimi K3 in 4-bit on a single 8xB300 Pod on Runpod

TL;DR

• Can you run Kimi K3 on a single node? Yes, on one 8xB300 pod, and you'll have about 800GB to spare for cache and context. That is the officially supported single-node floor. Nothing smaller works: 8xH100 (640 GB) and 8xB200 (1,536 GB) both fail to load it. You could probably get away with shaving off one or maybe even two of the B300s if your context needs aren't large, though.
• Do you need to quantize it? No. K3 was quantization-aware trained with MXFP4 weights and MXFP8 activations. The Hugging Face checkpoint is already 4-bit. There is no BF16 version to convert from.‍
• Why is a 4-bit 2.8T model 1.56 TB and not 1.4 TB? Only the routed MoE experts are MXFP4. Attention, KDA, Attention Residuals, the MoonViT-V2 vision encoder, and embeddings are BF16, plus per-group quantization scales. The repo averages ~4.49 stored bits per parameter.‍
• What serves it? vLLM via Docker only. vllm/vllm-openai:kimi-k3, CUDA 13. There is no working pip install path, because the integration depends on pre-release FlashInfer builds. The vLLM docker image can simply be dropped into the container path and you'll be good to go. SGLang does havea parallel day-0 recipe, but the scope of this guide only covers Kimi.

Working single-node config (verified on 8xB300, TP8, BF16 KV cache, 128K context): 

Drop this in for the container image: 

vllm/vllm-openai:kimi-k3

And this for the Container Start command: 

/workspace/models/Kimi-K3 --served-model-name kimi-k3 --host 0.0.0.0 --port 8000 --trust-remote-code --tensor-parallel-size 8 --load-format fastsafetensors --moe-backend flashinfer_trtllm   --max-model-len 128000 --max-num-seqs 16 --gpu-memory-utilization 0.90 --enable-prefix-caching --enable-auto-tool-choice --tool-call-parser kimi_k3 --reasoning-parser kimi_k3

Expose port 8000 as well to make sure vLLM can communicate with the outside world, and you'll be good to go.

What it costs: $59.12/hr on Secure Cloud, billed by the second, plus ~$100/month for a 1.8 TB network volume.

How long does it take? 20–30 minutes to download the weights, then about 30minutes for the engine to initialize before port 8000 responds.

Intro

Moonshot AI released the Kimi K3 weights on July 27, 2026: 2.8 trillion parameters, 16 of 896 experts active per token, Kimi Delta Attention with Attention Residuals, native vision through MoonViT-V2, and a 1M-token context window. It is the largest open-weight model anyone has shipped.

It is also the first frontier open-weight release where 4-bit isn't something you do to the model. It's what the model already is.

This guide takes you from an empty Runpod account to a working OpenAI-compatible endpoint on one 8xB300 pod. Budget about 60 minutes, most of it spent moving 1.56 TB of weights.

First: do you actually need a pod?

Runpod serves K3 two ways. The Public Endpoint is a way to simply run your prompts against a Runpod-managed API, which is great for smoke tests and exploration without the footprint of setting up an entire pod, and with the hardware requirements involved ithat is worth considering.

The Public Endpoint is OpenAI-compatible and needs no provisioning:

curl https://api.runpod.ai/v2/moonshot-kimi/openai/v1/chat/completions \
  -H "Authorization: Bearer $RUNPOD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "kimi-k3",
    "messages": [{"role": "user", "content": "What is Runpod?"}]
  }'

What each one is actually good at

Use the Public Endpoint when you're evaluating whether K3 fits your task, your traffic is spiky or low-volume, you want the full 1M context without a multi-node cluster, or you're prototyping an agent harness. It's also the honest answer for most production workloads under steady moderate load; per-token pricing on a shared cluster beats an underutilized $55/hour node by a wide margin. Depending on usage, there will be a point where the pod outscales the endpoint in terms of cost, if you can guarantee a high enough consistent workload for it (batching requests is a great example of this.)

Why 4-bit isn't a step

Most quantization guides follow the same arc: download BF16 weights, run a calibration pass, produce a smaller checkpoint, accept some quality loss. None of that applies here.

Moonshot applied quantization-aware training from supervised fine-tuning onward, using MXFP4 weights and MXFP8 activations. The 4-bit routed experts are the trained artifact, not a compressed copy of one. There is no BF16 K3 to quantize from, and community requantization has nowhere to go. Even Unsloth's 1-bit UD-IQ1_S GGUF lands at 594 GB.

So the deployment question isn't how do I get this to 4-bit. It's how do I fit a checkpoint that is already at the four-bit storage floor.

The arithmetic

The Hugging Face repo is 1,560.94 GB across 96 safetensors shards. Naive math on 2.8T parameters at half a byte per parameter suggests 1.4 TB, and the extra ~160 GB is not rounding error. Only the routed MoE experts are packed at four bits. The attention stack, the KDA layers, Attention Residuals, the vision encoder, and the embeddings are all BF16, and every group of quantized weights carries its own scales. The repository averages roughly 4.49 stored bits per parameter once everything is counted.

Against that, eight B300s give you 2,304 GB of HBM3E. That is a real fit, but it is not a generous one. That is why this guide lands on a 128K context window rather than the full 1M. Every token of context you reserve is a concurrent request you can't serve. Get correctness first, then push the window up or the batch size out once you've measured your own traffic. If you do need the full context then it's worth considering moving up to a cluster, though that will introduce its own level of overhead as the communication then needs to span multiple machines. One 8xB300 pod is the right shape for evaluation, agent-harness integration, prompt work, and low-concurrency internal serving.

Step 1: Provision the pod

The critical constraint: Docker is mandatory for K3. There is no working pip install vllm path, because the integration depends on pre-release FlashInfer builds. vLLM ships a K3-specific image, vllm/vllm-openai:kimi-k3, and it is CUDA 13 only.

On Runpod that's convenient rather than awkward, because a Pod is a container. You just make the K3 image the pod's image.

Create a custom template:

• Container image: vllm/vllm-openai:kimi-k3
• GPU: 8 × B300
• Container disk: 100 GB (just the image and scratch; .weights go on the volume)
• Volume: 1,800 GB minimum, mounted at /workspace
• Expose HTTP ports: 8000

Set the environment variables as well:

HF_TOKEN=hf_...
HF_HOME=/workspace/hf
HF_HUB_CACHE=/workspace/hf/hub
HF_XET_CACHE=/workspace/hf/xet
TMPDIR=/workspace/tmp
HF_HUB_ENABLE_HF_TRANSFER=1

You won't want to skip the Huggingface token; you'll get faster download speeds if you authenticate and when you're running a pod at $50/hr that's a real, tangible cost.

Setting only HF_HOME is not enough. The Xet backend keeps its own chunk cache that doesn't reliably follow HF_HOME, and some transfer paths stage through TMPDIR. Miss either and you will fill a small container disk while --local-dir points at a volume with terabytes free, resulting an out-of-space error that makes no sense until you realize the staging area and the destination are on different filesystems.

Step 2: Pull the weights

1.56 TB across 96 shards. Install the accelerated transfer backend, authenticate, and go:

pip install -U "huggingface_hub[hf_transfer]"

# Authenticate first — unauthenticated pulls are rate-limited and slower
hf auth login --token "$HF_TOKEN"
hf auth whoami   # confirm before starting a 1.5 TB transfer

export HF_HUB_ENABLE_HF_TRANSFER=1
hf download moonshotai/Kimi-K3 \
  --revision <pin-a-commit-sha> \
  --local-dir /workspace/models/Kimi-K3 \
  --max-workers 16

Run hf auth whoami before you start rather than after. An unauthenticated 1.5 TB pull will still succeed, just more slowly, and you'll only notice when the ETA is measured in hours, by which point you've burned real money watching it.

The two settings that actually move the needle here are HF_HUB_ENABLE_HF_TRANSFER=1 (the Rust transfer backend, which parallelizes chunks within each file) and --max-workers, which parallelizes across shards. Sixteen is a reasonable default on a pod with 256 vCPUs; going higher sometimes helps and sometimes just trips rate limiting.

Expect 20–60 minutes depending on the datacenter's egress path. Runpod doesn't charge for ingress, so the only cost here is time on the meter.

Verify before you launch:

du -sh /workspace/models/Kimi-K3
ls /workspace/models/Kimi-K3/*.safetensors | wc -l

You want roughly 1.5 TiB and 96 shards. This will ensure any quietly truncated downloads or other problems get caught now rather than later.

Step 3: Launch vLLM

Single node, no Ray, no cluster bootstrap. Tensor parallel across all eight GPUs. This is the configuration that came up clean on an 8xB300 pod:

vllm serve /workspace/models/Kimi-K3 \
  --served-model-name kimi-k3 \
  --host 0.0.0.0 \
  --port 8000 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --load-format fastsafetensors \
  --moe-backend flashinfer_trtllm \
  --max-model-len 128000 \
  --max-num-seqs 16 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --reasoning-parser kimi_k3

Run your actual server inside tmux or Jupyter. The web terminal is only intended for fast one-off tasks, and a dropped connection kills the server.

What each of the non-obvious flags is doing:

--load-format fastsafetensors: You are reading 1.5 TiB off disk, and time is money.

--moe-backend flashinfer_trtllm: LLM's guidance is flashinfer_trtllm for TP > 1 and deep_gemm_mega_moe for disaggregated or expert-parallel deployments. Single-node TP8 is the former.

--enable-prefix-caching:  this is off by default for K3. Its hybrid design needed a redesign of prefix caching to work over recurrent KDA state alongside conventional KV blocks. On agentic traffic with shared system prompts and tool definition, you'll want this turned on.

--reasoning-parser kimi_k3: K3 always thinks. There is no non-thinking mode. This routes the chain-of-thought into a separate reasoning_content field instead of leaving it inline in your response text.

--max-model-len 128000: deliberately not 1,048,576. With BF16 KV cache on eight cards, this is what comfortably fits alongside 16 concurrent sequences.

Startup takes longer than you think

The API server binds after engine initialization completes. Engine init means loading 1.5 TiB off disk, profiling, then capturing CUDA graphs across eight ranks, so you'll want to budget about a half-hour for this.

For that entire window, logs scroll busily and port 8000 refuses connections. There may be times where you won't see an update for several minutes. This reads as a hang and isn't one.

A successful load will end up with this when all is said and done:

(APIServer pid=932) INFO 08-27 21:16:26 [launcher.py:46] Route: /v1/completions/derender, Methods: POST
(APIServer pid=932) INFO 08-27 21:16:26 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=932) INFO:     Started server process [932]
(APIServer pid=932) INFO:     Waiting for application startup.
(APIServer pid=932) INFO:     Application startup complete.

Unless you specifically see an error, just be patient and give it time to do its thing; assume no news is good news until then.

Step 4: Verify

The service binds to 0.0.0.0:8000 inside the pod, and Runpod's HTTP proxy exposes it at:

https://<POD_ID>-8000.proxy.runpod.net

Smoke test:

curl https://<POD_ID>-8000.proxy.runpod.net/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "kimi-k3",
    "messages": [{"role": "user", "content": "Prove that the square root of 2 is irrational."}],
    "reasoning_effort": "high"
  }'

One proxy caveat: Cloudflare sits in that path and enforces a 100-second connection timeout. A reasoning_effort: max request on a hard problem will blow through that. Either stream the response, or expose port 8000 as a TCP port instead and connect to the direct IP of the pod instead (although bear in mind that this is also unencrypted traffic.)

Wrapping up

The interesting thing about K3 isn't that it's 2.8 trillion parameters. It's that Moonshot trained it quantization-aware and shipped the 4-bit weights as the artifact, which is what makes a single-node deployment possible at all. A year ago a model this size would have been a 64-GPU conversation before you could send it a single request. Now it fits on one pod you can rent by the second, and the binding constraint has shifted from "can I hold the weights" to "how much context can I afford alongside them."

That said, be honest about what one 8xB300 node is. It's an evaluation and integration platform: prove your agent harness against the real model, measure how your prompts behave at each thinking effort, test vision inputs, validate the tool-call plumbing. Moonshot's own production guidance is 64+ accelerators, and the gap between one node and that number is where concurrency and long-context headroom actually live. If your traffic is spiky or moderate, the Public Endpoint is very likely the better answer, until you have a reason for vLLM's more advanced features and then the pod setup is really going to shine.

Most of what will slow you down isn't the model. It's the 1.56 TB download landing on the wrong filesystem, an entrypoint that appends when you expected it to replace, and forty minutes of weight loading before a bad flag tells you it was bad. Those are the parts of this guide worth re-reading before you provision.

Tell us how it went

We'd like to know what you build with this, and what broke that we didn't warn you about.

• Discord: the #built-on-runpod channel is where people post what they're running. If your launch fails in a way this guide doesn't cover, that's the fastest place to get another set of eyes on it.
• r/Runpod: better for longer write-ups: benchmark numbers, cost breakdowns, config comparisons, and arguments about whether self-hosting was worth it.