The model works and the latency is fine. Then the invoice arrives. A frontier hosted model runs roughly $10-$15 (Sonnet/GPT Terra) to $25-$30 (Opus/GPT Sol) per million output tokens, and the math starts to hurt once you hit production-scale traffic. Self-hosting an open-weight model used to mean a steep, multi-step infrastructure lift. With vLLM running on a Runpod pod, it takes minutes.
This guide covers the pod-based deployment path: a full vLLM inference engine running as an OpenAI-compatible server on Runpod hardware, with working code you can run today. If you want the no-code serverless option (no SSH, no Docker), we cover that separately in Deploy vLLM on Runpod Serverless. That path is faster to start but trades away SSH control, persistent configuration, and the throughput tuning this guide walks through.
vLLM Explained: Why It's the Default LLM Inference Engine
vLLM is the default open-source LLM inference engine because it solves the one problem that makes naive serving collapse under load: GPU memory. It's built around PagedAttention, a memory-management technique that eliminates KV cache fragmentation and enables continuous batching. On the same hardware, that combination delivers well over an order of magnitude more throughput than serving requests one at a time.
vLLM achieves that by changing how the KV cache is allocated. The naive approach reserves one contiguous block of VRAM per sequence, sized for the longest output it might generate: most of that block sits unused, and the fragmentation it leaves starves other requests. PagedAttention drops the contiguous requirement. It stores the cache in fixed-size blocks handed out on demand, the way an operating system pages virtual memory, and reclaims them as sequences finish. Requests then pack into the same GPU far more densely, which is what lets vLLM fold new work into ongoing computation instead of finishing one request before starting the next. That efficiency is why vLLM became the default inference engine across the industry, from startups in production to research labs. Here's how to get it running on a Runpod pod.
Deploying vLLM on Runpod
Three steps get you from an empty Runpod console to a model serving live traffic.
Step 1: Spin up a Runpod pod for vLLM
Deploying vLLM starts with one pod, one container image, and one exposed port. Open Pods in the Runpod console, click + Deploy Pod, select your GPU, set the container image, and expose port 8000. Runpod generates a public proxy URL your application calls directly. By default, vLLM does not require an API key, so anyone with that URL can call your model. Add --api-key <your-secret> to the launch command in Step 2 before you send real traffic (full flag in the FAQ).
GPU selection comes down to model size and budget. Pricing moves often, so treat these as ballpark figures and check the Runpod pricing page for current rates:
- H100 SXM 80GB (~$2.99/hr): the highest throughput of these options for 8B through 13B models, or 70B models with tensor parallelism across two cards.
- A100 80GB (~$1.49/hr): a cost-effective step down for mid-sized models, and handles 70B with 2x A100.
- L4 24GB (~$0.39/hr): runs 7B-8B models in 4-bit quantization (AWQ/GPTQ), trading a small accuracy drop and higher per-token latency for the smaller VRAM footprint. Better suited to cost-sensitive workloads than maximum production throughput.
Under Container Image, use a pinned vLLM image tag rather than :latest. vLLM's throughput, default flags, and CLI behavior change between releases, and the flags and numbers in this guide assume a specific version. Check the current stable tag on Docker Hub, then set it explicitly:
vllm/vllm-openai:v0.25.1The image ships with vLLM and its CUDA runtime pre-installed, so there's nothing to install manually. Pinning also protects you from a silent breaking change the next time you redeploy the pod.
Set HTTP Port to 8000. Runpod creates a public proxy URL at https://{YOUR_POD_ID}-8000.proxy.runpod.net once the pod starts. That's the URL your applications call.
If you're loading a gated model like meta-llama/Llama-3.1-8B-Instruct, add an environment variable in the Environment Variables section before deploying:
HUGGING_FACE_HUB_TOKEN=hf_your_token_hereClick Deploy. The pod typically reaches running status in a minute or two. Model download follows: a few minutes for an 8B model, considerably longer for 70B, depending on network speed.
Step 2: Launch the vLLM OpenAI-compatible server
One command turns the pod into an OpenAI-compatible server. Once the pod is running, connect via the web terminal and start vLLM's built-in server. The --host 0.0.0.0 flag is required so Runpod's proxy can reach the process; without it, vLLM listens on localhost only and external requests never reach the proxy.
Connect via the Runpod web terminal (click Connect, then Web Terminal) and run:
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 8000 \
--served-model-name llama-3.1-8b \
--api-key your-secret-keyvllm serve is vLLM's current CLI; the older python -m vllm.entrypoints.openai.api_server --model <model> form still works if you prefer it. --served-model-name sets the name your API clients use in requests, and --api-key locks the endpoint behind a bearer token (drop it only for throwaway testing). For Qwen, Mistral, or a fine-tuned model, the command is identical: swap the model argument for your Hugging Face path or local directory.
A few other flags worth knowing (see vLLM's engine arguments reference for the full list):
# Limit context length (saves VRAM on memory-constrained GPUs)
--max-model-len 8192
# Pin the compute dtype (default "auto" follows the model config)
--dtype bfloat16
# Cap the fraction of GPU memory vLLM may use (weights + KV cache + activations)
--gpu-memory-utilization 0.90To persist a model across pod restarts and skip re-downloading it on every launch, attach a Runpod network volume when you deploy the pod (under Storage, create or select a network volume and mount it at /workspace), download the model there once, then pass --model /workspace/my-model on later launches.
Step 3: Test your vLLM endpoint on Runpod
Your pod now serves a fully OpenAI-compatible endpoint, so existing OpenAI SDK code works against it with a single base_url change. Point base_url from api.openai.com to your Runpod proxy URL, keep the same OpenAI Python SDK, and the rest of your application code stays as-is.
Replace {YOUR_POD_ID} with the ID from your Runpod console, and pass the key you set with --api-key:
curl https://{YOUR_POD_ID}-8000.proxy.runpod.net/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer your-secret-key" \
-d '{
"model": "llama-3.1-8b",
"messages": [{"role": "user", "content": "What is PagedAttention?"}],
"max_tokens": 200
}'The response comes back as standard OpenAI-format JSON, with latency depending on your workload. The same holds from the SDK, with no new library and no schema changes:
from openai import OpenAI
client = OpenAI(
api_key="your-secret-key", # matches the --api-key you launched vLLM with
base_url="https://{YOUR_POD_ID}-8000.proxy.runpod.net/v1",
)
response = client.chat.completions.create(
model="llama-3.1-8b",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain KV cache in three sentences."},
],
max_tokens=300,
temperature=0.7,
)
print(response.choices[0].message.content)For streaming responses (common in chat UIs), set stream=True exactly as you would against the OpenAI API:
stream = client.chat.completions.create(
model="llama-3.1-8b",
messages=[{"role": "user", "content": "Explain KV cache."}],
stream=True,
)
for chunk in stream:
print(chunk.choices[0].delta.content or "", end="")vLLM emits the same server-sent-event chunk format as the OpenAI API, so streaming works through the SDK unchanged.
vLLM Benchmark: Measuring Tokens per Second on a Runpod H100 SXM
Compatibility is only half the question; the other half is speed. On an H100 SXM 80GB pod, continuous batching and PagedAttention let a single card serve many concurrent requests, and the throughput you actually get depends on your model, sequence lengths, and traffic. So rather than trust a number from someone else's pod, measure your own. vLLM ships a throughput benchmark in its CLI:
vllm bench throughput \
--model meta-llama/Llama-3.1-8B-Instruct \
--input-len 512 \
--output-len 256 \
--num-prompts 1000Run it on the pod once the image is pulled, and it reports sustained tokens per second for your exact model, GPU, and sequence lengths. Whatever the absolute numbers, a few patterns hold:
- H100 SXM outpaces A100 80GB on the same model, mostly on memory bandwidth.
- An 8B model leaves a single card with plenty of headroom for batching, so throughput climbs with concurrency until the GPU saturates.
- A 70B model fits on one 80GB card only under FP8 or INT8 quantization, and with VRAM fully committed to a single instance there's little room for concurrent requests. That constraint is what tensor parallelism exists to relieve.
First-token latency is a separate axis from throughput. An 8B model on an H100 returns its first token quickly even when aggregate tokens per second is modest, so size your hardware for whichever metric your workload actually feels.
Multi-GPU Deployments: Tensor Parallelism for 70B Models
Models too large for a single GPU (Llama 3.1 70B, Qwen2.5 72B Instruct, any Mixtral variant) run across multiple cards with vLLM's --tensor-parallel-size flag, which splits weight tensors across GPUs using NCCL for inter-card communication.
The flag's value must evenly divide the model's attention-head count. Most 70B-class models use head counts divisible by 2, 4, and 8, which is why 2x, 4x, and 8x GPU configurations are the usual choices; an unsupported value fails at startup with a shape-mismatch error rather than a helpful message. All GPUs in the pod must also match in type and VRAM.
vllm serve meta-llama/Llama-3.1-70B-Instruct \
--tensor-parallel-size 2 \
--host 0.0.0.0 \
--port 8000 \
--api-key your-secret-keySelect a 2x H100 SXM configuration in the Runpod pod selector and set --tensor-parallel-size 2 to match; vLLM handles tensor placement from there. At 2x H100 SXM, Llama 3.1 70B reaches throughput comfortable for demanding production workloads, and scaling to 4x with --tensor-parallel-size 4 adds capacity close to linearly, up to the inter-card bandwidth limit. Since GPU-hours are the dominant cost driver, that scaling behavior feeds directly into the economics below.
The Cost Math: Self-Hosted vLLM on Runpod vs. OpenAI and Anthropic
For sustained, high-volume traffic, self-hosting on Runpod costs a fraction of managed-API pricing per token. A managed API charges a per-token markup that never goes away; a well-utilized pod charges a flat GPU-hour rate you amortize across every token it serves. Past a break-even volume, the pod wins and keeps winning.
The table assumes sustained throughput (many concurrent requests, not one idle user). Managed-API prices are list output-token rates; the self-hosted rows are estimates that move with GPU pricing:
*Estimated at sustained throughput (low thousands of tok/sec for 8B, several hundred for 70B) against current Runpod GPU pricing. Both the GPU rate and the achievable throughput vary, so treat these as directional and recompute against the Runpod pricing page for your own numbers.
The economics flip at low utilization. If you run occasional, low-volume requests on a pod that sits idle most of the day, you pay the hourly rate whether the pod is busy or not. Runpod bills per second, so you can stop the pod when you're done (click Stop on the pod in the console, or automate it through the Runpod API), but a stopped pod isn't running, and your endpoint is down until you restart it. A restart re-triggers model download unless the model lives on a network volume. For variable or spiky traffic, the serverless option is the better economic fit: you pay for active execution time instead of a standing hourly rate.
Self-hosting on a pod pays off on three fronts. It wins on sustained traffic that clears the break-even volume. It also wins when you need a fine-tuned or private model no public API offers, or when data-privacy rules keep prompts inside your own infrastructure.
Frequently Asked Questions
What GPU should I use to run vLLM on Runpod?
Match the card to the model: an 8B-13B model runs best on a single H100 SXM 80GB, an A100 80GB is the cheaper trade at lower throughput, and an L4 24GB fits 7B-8B models in 4-bit quantization for budget or low-volume use. The GPU-selection list in Step 1 has the current price ballparks.
Does vLLM on Runpod work with the OpenAI Python SDK?
Yes. vLLM exposes an OpenAI-compatible REST API at /v1/chat/completions and /v1/completions. Point base_url at your Runpod proxy URL and set api_key to the secret you launched vLLM with (any non-empty string works if you launched without --api-key). The same request payloads, streaming flags, and response objects work unchanged.
How long does it take to deploy vLLM on a Runpod pod?
Under fifteen minutes for an 8B model, end to end. The pod boots in under two minutes, the prebuilt image skips any install step, and the model download is the main variable, scaling with model size and network speed.
How do I run Llama 3.1 70B on Runpod with vLLM?
Run it across two GPUs: pick a 2x H100 SXM pod and add --tensor-parallel-size 2 to the launch command. The Multi-GPU section above covers the head-count divisibility rule and how throughput scales past two cards.
Is a Runpod pod or Runpod Serverless better for vLLM inference?
Choose pods for sustained, predictable traffic, or when you need SSH access, persistent configuration, or a private model. Choose serverless for variable or bursty traffic, where you're billed only while a request is running, not for idle GPU hours. The cost section above works through the break-even math.
Can I load a fine-tuned or private model with vLLM on Runpod?
Yes. Mount a Runpod network volume holding your weights at a path like /workspace/my-model, then pass --model /workspace/my-model to the server command. For gated Hugging Face models, set the HUGGING_FACE_HUB_TOKEN environment variable in the pod configuration before deploying. vLLM supports any architecture compatible with the Hugging Face transformers format.
How do I add API key authentication to vLLM on Runpod?
vLLM doesn't enforce API-key authentication unless you ask it to. Add --api-key your-secret-key to the launch command, and vLLM then requires an Authorization: Bearer your-secret-key header on every request. It's the minimum you want on any endpoint exposed through Runpod's public proxy. See vLLM's OpenAI-compatible server docs for TLS termination, CORS, and key-rotation options.
What's the difference between vLLM and Hugging Face TGI for self-hosted LLM inference?
vLLM and Hugging Face Text Generation Inference (TGI) are both production-grade inference servers with continuous batching and OpenAI-compatible APIs. vLLM generally leads on raw throughput benchmarks thanks to PagedAttention's memory efficiency, while TGI offers tighter Hugging Face Hub integration. Quantization-format support is broadly comparable between the two. Both run on Runpod pods as containerized deployments using the same pattern.
Wrapping Up
vLLM is the right default for self-hosted LLM inference right now, and it's the direct fix for the invoice problem that opened this guide. PagedAttention and continuous batching remove almost every reason to write custom inference code, and the OpenAI-compatible server keeps your existing SDK calls working. Getting it running on Runpod is a pinned Docker image and a single command, then a base_url swap in the client you already have. Past a modest traffic volume, it costs a fraction of per-token API pricing.
To spin up your first inference pod, head to the Runpod pricing page and pick a GPU. For batch or autoscaling workloads that don't justify a pod running around the clock, Runpod Serverless is the better starting point.
