The fastest GPU for LLM inference and the cheapest GPU for LLM inference are never the same card, and the gap is wider than most benchmark tables suggest. Across 21 GPUs running Qwen2.5-7B-Instruct on vLLM, per-user decode speed ranges from 17 to 272 tokens per second. Cost per million output tokens ranges from $0.64 to $17.40. Every figure below is a measured run on rented hardware, not a spec-sheet estimate.
Why tokens per second per user matters more than total throughput
GPU inference comparisons almost always lead with aggregate throughput: total tokens per second the server produces. Measured at a fixed request rate, it barely moves between cards.
Same test, same model, same engine. The left column is what an individual user experiences. The right is the aggregate figure implied by the cost data.
| GPU | Per-user decode | Implied aggregate |
|---|---|---|
| B200 | 272 t/s | 125.9 t/s |
| H200 | 195 t/s | 125.0 t/s |
| RTX 5090 | 101 t/s | 122.8 t/s |
| RTX 4090 | 61 t/s | 119.5 t/s |
| RTX A5000 | 43 t/s | 117.2 t/s |
| L4 | 17 t/s | 102.3 t/s |
Per-user decode spreads 16x. Aggregate throughput spreads 1.23x.
The reason is the part usually left out. These runs offer a fixed one request per second. At that rate the load generator, not the hardware, sets how many tokens leave the server. Every card is asked for the same work and nearly every card delivers it. What separates them is how fast any one user’s tokens arrive.
So aggregate throughput at a fixed request rate is closer to a measurement of the test harness than of the GPU. Quote it and you would conclude an L4 is 81% as good as a B200, while the person waiting on a response gets tokens 16 times slower.
This inverts under saturation. Push every card to its limit and aggregate throughput diverges sharply, because the larger cards hold larger batches. We have not published saturation runs, which is stated plainly below rather than glossed.
How these LLM inference benchmarks were measured
- Engine: vLLM
- Offered load: 1 request per second
- Measured: September 15, 2026
- Coverage: 15 text-generation models across 21 GPUs, seven workload shapes, 68 model-and-workload combinations
- Metrics: per-user decode rate, median time to first token, p95 time to first token, cost per million output tokens
Cost per million is derived from Runpod Secure Cloud on-demand rates on the measurement date. It is a calculation from those rates, not a quote.
Tokens per second by GPU: 21 cards on Qwen2.5-7B
The widest coverage in the dataset, 21 of 23 available GPUs on a single model.
| GPU | VRAM | Per-user decode | TTFT median | TTFT p95 | $ / 1M output |
|---|---|---|---|---|---|
| B200 | 180 GB | 272 t/s | 20 ms | 81 ms | $14.98 |
| B300 | 288 GB | 251 t/s | 48 ms | 324 ms | $17.40 |
| H200 | 141 GB | 195 t/s | 41 ms | 183 ms | $10.20 |
| H100 SXM | 80 GB | 165 t/s | 31 ms | 88 ms | $7.78 |
| H100 NVL | 94 GB | 160 t/s | 23 ms | 30 ms | $7.12 |
| H100 PCIe | 80 GB | 109 t/s | 43 ms | 49 ms | $6.52 |
| RTX 5090 | 32 GB | 101 t/s | 36 ms | 141 ms | $2.24 |
| A100 SXM | 80 GB | 96 t/s | 35 ms | 101 ms | $3.61 |
| A100 PCIe | 80 GB | 94 t/s | 39 ms | 49 ms | $3.61 |
| RTX Pro 6000 | 96 GB | 89 t/s | 35 ms | 84 ms | $4.76 |
| RTX 4090 | 24 GB | 61 t/s | 55 ms | 119 ms | $1.72 |
| RTX 6000 Ada | 48 GB | 58 t/s | 52 ms | 65 ms | $1.96 |
| RTX 3090 | 24 GB | 50 t/s | 91 ms | 103 ms | $1.18 |
| Pro 6000 MIG 48GB | 48 GB | 49 t/s | 62 ms | 72 ms | $2.57 |
| L40 | 48 GB | 48 t/s | 64 ms | 93 ms | $1.94 |
| L40S | 48 GB | 48 t/s | 66 ms | 82 ms | $2.58 |
| RTX A6000 | 48 GB | 44 t/s | 73 ms | 116 ms | $1.26 |
| RTX A5000 | 24 GB | 43 t/s | 81 ms | 95 ms | $0.64 |
| A40 | 48 GB | 35 t/s | 88 ms | 125 ms | $1.20 |
| Pro 6000 MIG 24GB | 24 GB | 25 t/s | 113 ms | 135 ms | $1.50 |
| L4 | 24 GB | 17 t/s | 175 ms | 232 ms | $1.33 |
Which GPU should you pick for LLM inference?
Cheapest GPU per million output tokens
The RTX A5000 serves this model at $0.64 per million output tokens, 27 times cheaper than the B300 at $17.40, and delivers 43 tokens per second per user, which is faster than most people read. For batch summarization, overnight document processing, or any job where nobody watches a cursor blink, the expensive card buys nothing.
The RTX A6000 at $1.26 and the A40 at $1.20 extend the same logic to 48 GB, where 27B-class models start to fit.
Fastest GPU for time to first token
The B200 returns a first token in 20 ms and decodes at 272 tokens per second. The result worth noticing is the H100 NVL: 160 tokens per second with a p95 time to first token of 30 ms, the tightest tail in the table. The B300’s p95 is 324 ms, more than ten times worse, despite a faster median.
If a person is waiting on your product, p95 decides whether it feels reliable. Median hides the bad requests.
Best value: RTX 5090 and RTX 4090
The RTX 5090 returns 101 tokens per second at $2.24 per million: 37% of a B200’s decode speed for 15% of its cost per token. The RTX 4090 at $1.72 per million is the same trade one step down.
Neither card holds a 27B model at BF16. That constraint usually decides this tier, not speed.
How much VRAM do you need for LLM inference?
VRAM is a hard gate. Speed only matters among cards the model fits on.
| Model size and precision | Minimum practical VRAM | Cards worth testing |
|---|---|---|
| 7B–9B, quantized | 24 GB | RTX A5000, RTX 3090, RTX 4090, L4 |
| 7B–14B, FP8 | 32–48 GB | RTX 5090, A40, L40, RTX 6000 Ada |
| 27B–35B, FP8 | 48–80 GB | RTX 6000 Ada, A100 SXM, RTX Pro 6000 |
| 27B+, BF16 | 80 GB+ | H100 SXM, H200, B200, B300 |
Sparse and dense models of the same nominal size behave very differently. Qwen3.6-27B (FP8) tops out at 105 tokens per second on a B200. Qwen3.6-35B-A3B (FP8), a larger sparse model, reaches 260 on a B300 and still manages 99 on an L40 at {{gpu:l40}}/hr. A mixture-of-experts model rewards cheap VRAM more than it rewards memory bandwidth.
What these benchmarks do not cover
- No saturation runs. Everything here is one request per second. Cards with large VRAM and high bandwidth pull further ahead as concurrency rises. Treat the cheap cards’ cost advantage as a best case.
- Nothing above 35B parameters. The largest benchmarked models are Qwen3.6-35B-A3B (FP8) and Gemma 4 31B IT. Sizing above that is memory math.
- vLLM only. TensorRT-LLM and SGLang will shift these rankings, in some cases substantially.
- Training is not covered. These are inference benchmarks. For training and fine-tuning hardware, see the best GPU for AI training guide.
- Cost per million is a snapshot from Secure Cloud rates on September 15, 2026.
Common questions about the best GPU for LLM inference
What is the best GPU for LLM inference?
It depends which axis you are optimizing. For lowest cost per token, an RTX A5000 serves Qwen2.5-7B at $0.64 per million output tokens. For lowest latency, a B200 decodes the same model at 272 tokens per second with a 20 ms first token. Both figures come from the same test on the same model.
How many tokens per second does an H100 do?
On Qwen2.5-7B-Instruct at one request per second, an H100 SXM delivers 165 tokens per second per user and an H100 PCIe 109. The H100 SXM costs $7.78 per million output tokens at that rate, among the highest in the test.
How many tokens per second does an RTX 4090 do?
61 tokens per second per user on the same model and test, at $1.72 per million output tokens. An RTX 5090 returns 101 at $2.24, and an RTX 3090 returns 50 at $1.18.
How much faster is an H100 than an RTX 4090 for inference?
Roughly 2.7 times on per-user decode, 165 tokens per second against 61. It also costs about 4.5 times more per million output tokens, $7.78 against $1.72. Faster per user, worse per dollar.
Does aggregate throughput tell me which GPU to pick?
Not at a fixed request rate. Across these runs aggregate throughput varied 1.23x between the fastest and slowest card while per-user decode varied 16x, because the offered load rather than the hardware set the total. Aggregate throughput becomes meaningful only in saturation tests.
What is a good time to first token?
Under 100 ms is imperceptible in a chat interface, and 19 of the 21 cards clear that on median. Only the Pro 6000 MIG 24GB at 113 ms and the L4 at 175 ms do not. Watch p95 rather than median: the H100 NVL holds 30 ms at p95 while the B300 reaches 324 ms, so the card with the faster median can deliver the worse experience.
Can I run a 70B model on a single GPU?
Not at BF16. A 70B model needs roughly 140 GB for weights alone. Only the B200 at 180 GB and the B300 at 288 GB leave meaningful room beyond that; the H200’s 141 GB holds the weights and nothing else, which is not a working configuration once KV cache is accounted for. Quantized to FP8 it fits comfortably on an H200 or larger. This dataset does not benchmark models that size.
Compare these figures interactively, across all four workload types, on the GPU Compare tool.
Author profile: The Runpod Team
