gpt-oss-120b is OpenAI's open-weight reasoning model, released 5 August 2025 under the Apache 2.0 license. It has 117B total parameters but activates only 5.1B per token, and it ships natively quantized in MXFP4, so the weights occupy roughly 60 GB and the model fits on a single 80 GB GPU.
Apache 2.0 matters here. You can use it commercially, modify it, and self-host it without a separate license negotiation. That is not true of every open-weight model, and it is the reason this one shows up in production stacks rather than only in experiments.
There are two ways to run it on Runpod, and the right one depends entirely on your volume.
Two ways to run gpt-oss-120b
Call the hosted endpoint. Runpod runs gpt-oss-120b as a Public Endpoint at $10.00 per 1M tokens. No container, no GPU decision, no deployment. It is OpenAI-compatible, so it drops into existing code by changing a base URL.
Self-host it on your own GPU. You rent a card, serve the model with vLLM, and pay per second for the hardware rather than per token. This wins once volume is high enough that a card running most of the day costs less than the tokens you would push through the endpoint.
The crossover is arithmetic, not preference. Work out your monthly token volume, multiply by the endpoint rate, and compare it against the hourly cost of a card running as much as you actually need it. Below that line the endpoint is cheaper and far less work. Above it, self-hosting is.
The hosted endpoint
The fastest path from nothing to a working request. The endpoint is OpenAI-compatible, so the standard client works with a changed base URL:
from openai import OpenAI
client = OpenAI(
api_key=RUNPOD_API_KEY,
base_url="https://api.runpod.ai/v2/gpt-oss-120b/openai/v1",
)
response = client.chat.completions.create(
model="gpt-oss-120b",
messages=[
{"role": "user", "content": "Explain quantum entanglement simply."}
],
max_tokens=512,
)
Streaming works by adding stream=True. There is also a native endpoint at https://api.runpod.ai/v2/gpt-oss-120b/runsync taking an input object, and a playground in the Runpod Hub if you want to try the model before writing any code.
Billing is $10.00 per 1M tokens as of 25 August 2026, so 10,000 tokens costs $0.10 and 100,000 costs $1.00. Because you are billed per token, an idle application costs nothing.
What you are actually deploying
The mixture-of-experts design is the whole story. A dense 117B model would need several GPUs and would cost accordingly. Because only 5.1B parameters are active on any given token, gpt-oss-120b runs at a speed closer to a small model while answering like a much larger one.
The model supports three reasoning efforts, set in the system message. They trade latency against accuracy, and the gap is large enough to be a deployment decision rather than a tuning detail. These are vLLM's reproduced scores, which vLLM notes vary across runs:
If your workload is classification or extraction, low effort will be faster and cheaper and you will not miss the difference. If it is maths or multi-step reasoning, high effort nearly doubles the AIME score.
Which GPU to rent for gpt-oss-120b
If you have decided to self-host, this is the decision that follows.
Start on an H100 SXM. It is the configuration vLLM's own recipe is tuned for, and one card is enough.
Move to H200 when concurrency is the constraint rather than speed. On an 80 GB card the MXFP4 weights leave roughly 13 GB for KV cache, which is where your concurrent request ceiling comes from. H200's 141 GB is not about running a bigger model, it is about serving more people at once.
Move to B200 when you want Blackwell's native MXFP4 tensor cores, which use a different and faster MoE kernel than Hopper does.
Choose A100 PCIe if cost matters more than throughput. It works, and it is less than half the price of an H100 SXM, but it runs a slower kernel path.
Deploying gpt-oss-120b with vLLM
Pull the official image and serve the model:
docker run --gpus all -p 8000:8000 --ipc=host \
vllm/vllm-openai \
--model openai/gpt-oss-120b
One gotcha worth knowing before you hit it. On an H100 at tensor-parallel size 1, the default memory settings will cause a CUDA out-of-memory error. Raise the memory utilization and lower the batched token count:
vllm serve openai/gpt-oss-120b \
--gpu-memory-utilization 0.95 \
--max-num-batched-tokens 1024
On Blackwell, set the FlashInfer MoE kernel before launching:
export VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8=1
For a production configuration, vLLM publishes separate config files for Hopper and Blackwell. Both set no-enable-prefix-caching, max-cudagraph-capture-size: 2048, max-num-batched-tokens: 8192 and stream-interval: 20. The Blackwell file adds kv-cache-dtype: fp8.
Tensor parallelism is the main lever after that. Set it to 1 for maximum throughput per GPU, and to 2, 4 or 8 when you care more about per-user latency than about cost per token.
The harmony format is not optional
gpt-oss models were post-trained on OpenAI's harmony prompt format. Serving them without it produces degraded output, and this is the single most common reason a first self-hosted deployment disappoints. The hosted endpoint handles this for you.
If you are serving it yourself, vLLM handles harmony on three endpoints:
/v1/responsesrenders harmony through theopenai-harmonylibrary and supports built-in tools. OpenAI recommends this endpoint./v1/chat/completionsgives the familiar interface, returns reasoning and final text separately, and acceptsinclude_reasoning: falseif you do not want the chain of thought back./v1/completionsis raw input and output with no template rendering, which means you are responsible for harmony yourself.
For function calling, launch with --tool-call-parser openai --enable-auto-tool-choice. Only tool_choice="auto" is supported.
One thing to get right before you ship: do not display the chain of thought to end users. OpenAI applied no direct supervision to it, so it can contain content that the final answer correctly excludes.
Serverless or a pod
If you self-host, you still have a choice of shape.
Use a pod when you are evaluating the model, benchmarking configurations, or running a job with steady load. You pay for the hour, you keep the container, and you can iterate on the vLLM flags without redeploying.
Use Serverless when traffic is uneven. The model loads once per worker and scales to zero between requests, so a workload that runs a few hours a day does not pay for the other twenty.
The break-even is utilization, not preference. Above roughly half the day busy, a pod is usually cheaper. Below it, Serverless usually is.
Going faster on the same GPU
Once a gpt-oss-120b endpoint is serving real traffic, the next lever is not a bigger card. It is the serving stack.
Runpod Overdrive optimizes any vLLM-compatible model running on Runpod Serverless, which includes this one. Runpod publishes per-model figures, and gpt-oss-120b is one of the four models benchmarked:
- Up to 2.33x improvement in inter-token latency
- Up to 1.72x higher throughput
- Up to 1.66x improvement in end-to-end latency
- Up to 3.28x faster time to first token on prefill-heavy workloads
Those figures are measured on an H100 SXM 80GB on Runpod Serverless, at what Runpod describes as a near-lossless eval score. Read them as a ceiling rather than a guarantee: the gain depends on your context length and traffic pattern, which is why the process starts with benchmarking your existing endpoint.
It is worth knowing about at the point where you are weighing an H200 or a B200 purely for speed. Optimizing the stack on the card you already have is the cheaper experiment, and Overdrive is priced so that you pay nothing if it does not beat your measured baseline.
When not to self-host
If your volume is low, use the hosted endpoint. Standing up a 117B model to answer a few hundred requests a day costs more than the tokens would, and you inherit the operations. This is the most common mistake with a model this size.
If gpt-oss-20b is good enough, run that instead. It fits in 16 GB, which means an RTX A5000 at {{gpu:rtx-a5000}}/hr on Secure Cloud rather than an H100 PCIe at {{gpu:h100-pcie}}/hr. Test the 20b first. If it clears your quality bar, the 120b is a large cost increase for nothing.
If you need multimodal input, this is the wrong model. gpt-oss is text-only. No image, audio or video input at any size.
If your compliance regime requires physical custody of the hardware, renting will not satisfy it regardless of how the tenancy is configured.
Frequently asked questions
Can I use gpt-oss-120b without deploying it?
Yes. Runpod runs it as a Public Endpoint at $10.00 per 1M tokens, fully compatible with the OpenAI API. Point the OpenAI client at https://api.runpod.ai/v2/gpt-oss-120b/openai/v1 and it works with your existing code. There is also a playground in the Runpod Hub for testing before you write anything.
What are the GPU requirements for gpt-oss-120b?
A single 80 GB GPU. The MXFP4 weights occupy roughly 60 GB, leaving about 13 GB for KV cache on an 80 GB card. H100, H200, B200 and A100 all work. On Runpod that starts at {{gpu:a100-pcie}}/hr for an A100 PCIe and {{gpu:h100-pcie}}/hr for an H100 PCIe on Secure Cloud.
How much VRAM does gpt-oss-120b need?
Around 60 GB for the weights themselves. Plan for 80 GB total so there is room for KV cache. If you want meaningful concurrency, a 141 GB H200 gives you roughly ten times the cache headroom.
Can I make gpt-oss-120b faster without renting a bigger GPU?
Yes, by optimizing the serving stack. Runpod Overdrive optimizes vLLM-compatible models on Runpod Serverless and publishes gpt-oss-120b figures of up to 2.33x better inter-token latency, up to 1.72x higher throughput and up to 3.28x faster time to first token on prefill-heavy workloads, measured on an H100 SXM 80GB. Actual gains depend on your context length and traffic pattern.
Should I use the hosted endpoint or self-host?
It comes down to volume. The endpoint bills per token and costs nothing when idle, which suits variable or low traffic. Self-hosting bills per second for a card, which wins once you are pushing enough tokens that a mostly-busy GPU costs less than the equivalent token spend. Estimate your monthly tokens and compare the two directly.
Can I run gpt-oss-120b on a consumer GPU?
Not on a single one. The weights alone exceed any consumer card's VRAM. gpt-oss-20b needs only 16 GB and runs comfortably on consumer hardware.
Is gpt-oss-120b free to use commercially?
Yes. It is released under Apache 2.0, which permits commercial use, modification and redistribution. Read the license yourself before you build on it, but there is no separate commercial agreement to negotiate.
How many parameters does gpt-oss-120b have?
117B total, with 5.1B active per token. It is a mixture-of-experts model with 128 experts across 36 layers, four of which are active for any given token.
What context length does gpt-oss-120b support?
128k tokens natively.
What is the difference between gpt-oss-120b and gpt-oss-20b?
Size and hardware. The 120b has 117B parameters and needs 80 GB; the 20b has 21B and needs 16 GB. Both use the same harmony format, both support 128k context, and both are Apache 2.0. The 120b scores higher on reasoning benchmarks; the 20b is far cheaper to serve.
Why is my gpt-oss output garbled?
Almost always the harmony format, and it only affects self-hosted deployments. Use /v1/responses or /v1/chat/completions rather than /v1/completions, or render harmony yourself. If output is still wrong, check that no conflicting Triton package is installed and that your CUDA version is 12.8 or higher.
