The RTX 5090 Is Here: Serve 65,000+ Tokens Per Second on Runpod

In April 2025, Runpod added the NVIDIA RTX 5090 to its GPU lineup. We benchmarked it for real-time LLM inference on small and mid-sized models, the kind used in high-concurrency chatbots, inference APIs and multi-model backends.

Small Models, Huge Gains

In our internal benchmarks, the 5090 posted high throughput on small language models. Qwen2-0.5B achieved over 65,000 tokens per second and 250+ requests per second at 1024 concurrent prompts—a 2–3x leap compared to previous-gen cards. Similarly, Phi-3-mini-4k-instruct delivered 6,400 tokens/sec and ~25 requests/sec under the same conditions.

Even at lower batch sizes, both models maintained high throughput, with Qwen2-0.5B consistently outperforming Phi-3 in raw speed. It’s worth noting that these numbers represent high-concurrency performance, which is ideal for inference APIs and production deployments. As expected across the industry, single-prompt throughput will be lower.

Efficiency and Scaling

All tests were run using vLLM, a high-performance inference engine optimized for large-batch and low-latency workloads. During our benchmarks, CPU usage averaged just 3.2%, confirming that the system is well-balanced and GPU-bound. VRAM usage sat at 95%, showing the 5090 was fully loaded with memory-bound tasks.

What really stands out is the linear scaling we observed as batch sizes increased—particularly with Qwen2-0.5B. Even at 1024 concurrent prompts, there was no significant drop-off in efficiency.

How We Tested

The benchmark setup was simple but representative of typical chat and instruction-following use cases. We used:

• vLLM as the backend (tensor parallelism = 1, single GPU)
• 128 input + 128 output tokens per prompt
• Default precision (dtype: auto, no quantization)
• RTX 5090 Config: gpu_memory_utilization=0.95
• And the GPU utilized around 18 GB of its available 32 GB of VRAM

This means there’s still room to experiment—with quantized models, larger architectures, or even multi-model deployments on the same GPU.

What This Means for You

In these tests, the RTX 5090 served small models at more than 250 requests per second while using 18 GB of its 32 GB of VRAM, which left room for larger models or several models on the same GPU.

For new workloads on Runpod that need more than 24 GB of VRAM, the RTX A6000 and L40S offer 48 GB. Both are slower than the 5090, so the numbers above won't carry over. Benchmark your own model before you size a deployment.

To run a similar test, start with our inference containers or the vLLM template.

‍

Author profile: Alyssa Mazzina