What is Kimi K3, in one paragraph?
Kimi K3 is Moonshot AI's open weight flagship: a native multimodal mixture-of-experts model with 2.78 trillion total parameters and 104 billion activated per token. It uses a hybrid attention stack (Kimi Delta Attention interleaved with Gated MLA), Attention Residuals for cross-depth information flow, and Stable LatentMoE with 896 routed experts (16 active per token). Context is up to 1 million tokens, weights ship in MXFP4, and reasoning is always on with a configurable effort level. Moonshot reports roughly 2.5x the scaling efficiency of K2.
Is Kimi K3 open source, and is it free to use?
Yes, with a caveat about what "free" means. Moonshot published the full K3 weights on Hugging Face under a custom Kimi K3 License, an MIT-derived permissive license. You can download, run, modify, fine-tune, distribute, and deploy it commercially at no license cost. Two conditions apply only at large scale: if you operate a Model-as-a-Service business with more than $20 million in revenue over any consecutive 12 months you must sign a separate agreement with Moonshot, and any product built on K3 with more than 100 million monthly active users or more than $20 million in monthly revenue must display "Kimi K3" in its interface. Internal use, and use through Moonshot's official products or certified inference partners, are exempt. For nearly everyone below those thresholds it is effectively free and open to use.
Free to obtain is not the same as free to run. There is no per-token license fee on the weights, but serving K3 costs compute: self-hosting means paying for Blackwell GPU time, and the hosted Runpod endpoint is billed per token.
What does the architecture actually look like?
Three ideas do most of the work.
- Hybrid attention. Each block is three KDA layers followed by one Gated MLA layer, a 3:1 ratio, repeated through the backbone, with a final Gated MLA layer so the last layer is always global. Across 93 layers that lands at 69 KDA and 24 MLA layers. KDA is a linear-attention mechanism (delta-rule recurrence with a channel-wise forget gate), so it carries a fixed-size recurrent state rather than a growing KV cache. The periodic MLA layers restore full global attention with a compressed latent KV. Positionally, K3 uses no explicit positional encoding (NoPE): position is encoded implicitly through KDA's gating and decay, which is how it extrapolates to 1M tokens without RoPE rescaling.
- Attention Residuals (AttnRes). Instead of only accumulating a sequential residual, each layer can attend over the embedding plus preceding block outputs through learned pseudo-queries. K3 uses the block variant, partitioning layers into blocks of 12 (about 8 blocks), which drops the overhead from O(Ld) to O(Nd) and bounds the inference-time state.
- Stable LatentMoE. The router projects each token down to a latent dimension of 3,584 (half the 7,168 hidden size) before dispatch, which cuts the communication and expert-weight traffic that normally grows with routing multiplicity. It runs 896 routed experts with 16 active and 2 shared experts, and adds normalization, a SiTU-GLU activation, and Quantile Balancing to keep training stable at that sparsity.
Native vision is handled by MoonViT-V2 (a 401M, 27-layer ViT, patch size 14), trained jointly with text from the start rather than bolted on afterward.
Kimi K2 vs Kimi K3, spec by spec
The short read: K2 was a pure-MLA FP8 MoE that fit one 8xH200 node. K3 roughly triples the parameters, doubles the active parameters and expert count, swaps most of the attention stack to linear KDA, extends context 8x to 1M, adds native vision, and ships in MXFP4. It is a much different serving profile, not a bigger version of the same one.
What is KDA, and why does a serving engineer care?
Kimi Delta Attention is the delta-rule linear-attention recurrence with a per-channel forget gate. Queries and keys go through a short convolution, Swish, and L2 normalization, and each KDA layer maintains a recurrent state of shape dk by dv that is updated per chunk (parallel within a chunk, recurrent across chunks).
The serving consequence is memory. In a standard transformer the KV cache grows linearly with context, so at 1M tokens it dominates VRAM. With K3, three out of every four attention layers are KDA and carry a constant-size state instead of a growing cache. Only the 1-in-4 Gated MLA layers hold a global KV, and that one is a compressed MLA latent. So the KV footprint at long context is a small fraction of what a full-attention model of this size would need. That is what makes a 1M window practical and long-context decode much faster, and it shifts the binding constraint at deployment from KV cache to simply fitting the weights.
It also enables state-aware prefix caching: because the KDA layers summarize history into a recurrent state, that state can be cached and reused across requests that share a prefix, which the K3 systems stack is built around.
Why MXFP4, and what does it mean for precision and hardware?
K3 does not post-hoc quantize a BF16 checkpoint. It runs MXFP4 quantization-aware training through the entire post-training stage, SFT and RL, so the model adapts to the low-precision numerics and there is no train-versus-inference mismatch. Only the MoE expert weights (the bulk of the parameters) go to MXFP4, activations are MXFP8, and the non-expert parts (attention projections, latent MoE projections, shared experts, routers) stay in higher precision.
Two practical implications. First, the released MXFP4 weights are roughly 1.56 TB on disk, versus about 5.6 TB for BF16, and that is the format you serve. Second, MXFP4 is native on NVIDIA Blackwell and AMD MI355-class hardware, so those are the sensible targets. On older parts without native MXFP4 you fall back to slower paths.
What does it take to serve K3 today?
The current path is vLLM through the official Docker image, vllm/vllm-openai:kimi-k3. The KDA kernels depend on pre-release libraries including FlashInfer, so the day-one route is the image rather than a plain pip install. A single-node serve is tensor parallel across 8 GPUs:
vllm serve moonshotai/Kimi-K3 \
--tensor-parallel-size 8 \
--trust-remote-code \
--load-format fastsafetensors \
--enable-prefix-caching \
--enable-auto-tool-choice \
--tool-call-parser kimi_k3 \
--reasoning-parser kimi_k3To turn on the speculative-decoding draft (see the next question), add the EAGLE-3 draft config:
--speculative-config
'{"model":"Inferact/Kimi-K3-DSpark","method":"dspark","num_speculative_tokens":7,"attention_backend":"FLASHINFER_MLA"}'Two operational notes. Pin the image to the exact published tag rather than a floating one, because the tooling for a brand new architecture moves fast in the first days. And for the two-node 16xB200 option, multi-node tensor parallelism needs a high-bandwidth interconnect between nodes, which is what an Instant Cluster provides (up to 3200 Gbps on Blackwell) and a pair of unlinked pods does not. Validate a small serve before committing a full node.
Hardware, from the MXFP4 footprint: where you can get the capacity, a single node of 8xB300 (288 GB each, about 2.3 TB) fits the weights plus KV and is the simplest option. Newest-generation Blackwell supply fluctuates, so 16xB200 across two nodes is the alternative. Check current availability for the configuration you want before planning around it.
What is the speculative-decoding story?
K3 is pre-trained with a multi-token-prediction layer whose structure mirrors a backbone block, and that MTP layer is fine-tuned into an EAGLE-3-style single-layer draft model. The draft fuses low, mid, and high level features from the 1st, 4th, and final AttnRes blocks and is trained by directly maximizing the acceptance rate rather than a KL surrogate. In practice a first-class speculative-decoding path ships with the model (the DSpark draft in the config above), and it is the main lever for pushing decode throughput on a serving deployment.
What are the operational gotchas?
Three things to plan for before K3 hits production:
- Pre-release dependencies. The kimi-k3 image pulls pre-release libraries, FlashInfer among them. Pin the exact image, and expect faster-than-usual churn in the first weeks. Keep a known-good tag you can roll back to.
- Cold start. A 1.56 TB model is not a fast cold start. The first load has to pull and place well over a terabyte of weights, so plan for a meaningful warm-up window and keep workers warm for latency-sensitive traffic. Caching the weights on a network volume in the same region avoids paying the download repeatedly across restarts.
- Always-on reasoning. K3 always emits a reasoning stream, and those tokens count against max_tokens. Set a generous max_tokens and read reasoning_content, or a small budget gets consumed by thinking and returns an empty answer. This is the single most common integration surprise.
How do I call the Kimi API on Runpod without provisioning GPUs?
Runpod runs an OpenAI-compatible API Public Endpoint for the Kimi models. Set the base URL to https://api.runpod.ai/v2/moonshot-kimi/openai/v1, authenticate with your RUNPOD_API_KEY, and set model to kimi-k3. Any OpenAI-compatible client works. K3 returns the reasoning stream as reasoning_content next to the answer, and reasoning_effort accepts low, high, or max. The same endpoint serves kimi-k2.6 and kimi-k2.7, and you can switch between them with the model string. Billing is per token, with input and output priced separately; see the endpoint docs for the current rates. Rate limits apply per account, so if you are load-testing, expect throttling rather than linear scaling on a single key. This is the fastest way to try K3 and gauge the right fit for spiky or low-volume traffic. The endpoint, its models, and current pricing are documented at docs.runpod.io/public-endpoints/models/moonshot-kimi.
When should I self-host instead?
Self-hosting flips the economics from per token to paying for the GPUs by the time you run them, so it pays off once volume is steady and you can keep the hardware busy. On Runpod that means an 8xB300 Pod for a single-node serve or an Instant Cluster for multi node, running the K3 Docker image against the weights. You own the endpoint and pay for the compute you use (Pods bill per minute while running), so cost per token becomes a utilization question you control.
How do I work out the self-host breakeven?
The math is one line. Effective cost per million output tokens equals the hourly cost of the node divided by sustained output tokens per second, divided by 3,600, times one million:
$/1M output = (node $/hr) / (sustained output tok/s) / 3600 * 1,000,000
Plug in the current node rate from runpod.io/pricing (for example, an 8xB300 node is 8 times the single per-GPU B300 rate). The one number you must measure rather than assume is sustained output tokens per second on your own traffic, because it depends on batch size, context length, and whether the DSpark speculative path is on. Run K3 on representative traffic, read the sustained tok/s, put it in the formula, and compare against the hosted endpoint's per-token price. The crossover is where a busy self-hosted node beats per-token pricing.
How Runpod serves K3
Runpod exposes K3 through two serving surfaces on the same platform.
The hosted path is a Runpod Public Endpoint: the public, OpenAI-compatible Kimi API described above (https://api.runpod.ai/v2/moonshot-kimi), billed per token. Nothing to provision, you send OpenAI-format requests and get K3 back with the reasoning stream in reasoning_content. It is the low-friction option for trying the model and for spiky or low-volume traffic.
Self-hosting runs the model on GPUs you control. You deploy the official vLLM kimi-k3 Docker image onto Runpod Blackwell capacity: a Pod for a single 8xB300 node with tensor-parallel-size 8, or an Instant Cluster when you want multiple nodes such as 16xB200. Runpod supplies the GPU capacity and orchestration (deploy in minutes, autoscale; Pods bill per minute, Instant Clusters per GPU-hour), and vLLM supplies the serving engine.
On that self-hosted stack you are running K3's serving-oriented design directly. The MXFP4 expert weights execute on Blackwell's native MXFP4 path. The hybrid attention keeps the KV cache small at long context, because only the periodic Gated MLA layers hold a global cache while the KDA layers carry a constant recurrent state. State-aware prefix caching reuses that KDA state across requests that share a prefix, and the EAGLE-3 draft model derived from K3's MTP layer drives speculative decoding. Those are the levers that set throughput and cost per token, and on a self-hosted deployment they are yours to tune.
When should I reach for K3 versus a smaller model?
K3 is built for frontier work: long-horizon coding and agentic loops over hundreds of tool calls, reasoning across very long context, and multimodal input. The MoE sparsity means each token only touches about 104B of the 2.8T parameters, so quality per step is high, but you still pay to hold the full 1.56 TB resident. If the job does not need 1M context, vision, or top-end reasoning, a smaller model is cheaper to run and fits far more common (and less expensive) hardware. Match the model to the task rather than defaulting to the largest model and most powerful hardware available.
Run K3 on Runpod
Try it in minutes on the hosted Kimi API endpoint, then benchmark a self-hosted 8xB300 Pod or a Cluster against your own traffic to find your real cost per token. Start at runpod.io, or talk to the team if you want help sizing a K3 deployment.
References
- Kimi K3 technical report: https://github.com/MoonshotAI/Kimi-K3/blob/main/k3_tech_report.pdf
- Kimi K3 model weights on Hugging Face: https://huggingface.co/moonshotai/Kimi-K3
- Moonshot Kimi K3 blog: https://www.kimi.com/blog/kimi-k3
- Runpod public endpoint docs for the Kimi models: https://docs.runpod.io/public-endpoints/models/moonshot-kimi
