Cut your vLLM cold starts from 5 minutes to 90 seconds on Runpod Serverless

Cold starts are the tax you pay for autoscaling, and on an LLM endpoint you pay it in two places.

The infrastructure cold start is the platform provisioning compute, attaching storage, and pulling your container image over the network. The vLLM cold start is everything after the container is running: loading model weights into GPU memory, running torch.compile, and capturing CUDA graphs. The second one is where the money goes. For a 30 GB model, vLLM initialization alone can top five minutes, and whoever sent the request waits through all of it.

We ran a controlled A/B test on Runpod Serverless to find out how much of that vLLM phase is unavoidable. About a quarter of it. Four configuration changes took a worker serving a 32B FP8 model on two H200s from a 324-second cold start to 91 seconds, a 3.5× improvement, with no custom code and no change to request-time latency.

The anatomy of a cold start

When a fresh vLLM worker boots, the clock breaks roughly into four steps. The first belongs to the infrastructure phase, the rest to vLLM. 

1. Image pull. Seconds if the image is cached on the host, 30 to 90 seconds if it is not. Nothing in the config below touches this, but it is inside the numbers we report, so it is worth knowing it is there.
2. Weight load. Reading safetensors shards and copying them to GPU. The default loader memory-maps files lazily, which on network storage means the read cost lands during materialization. For a 32 GB model this can take several minutes.
3. torch.compile and CUDA-graph capture. vLLM's default compile mode builds kernels and captures CUDA graphs for a list of batch sizes. On a cold worker this runs from scratch, every time. We measured roughly two to four minutes here.
4. KV cache allocation and warmup. Usually the smallest slice.

Steps 2 and 3 dominate, and both produce artifacts that survive being written to disk. That is what the four changes below exploit.

The test

Model: Qwen/Qwen3-32B-FP8 (roughly 32 GB of weights), tensor_parallel_size=2 on 2× NVIDIA H200 (141 GB).

‍Image: runpod/worker-v1-vllm:v2.25.0 (vLLM 0.27.0), weights on a network volume.

‍What we are measuring. The job's delayTime, the interval between job submission and job pickup by a worker. When the endpoint has no running workers, that interval contains the entire boot: image pull, weight load, compile, graph capture, and warmup. It is an end-to-end number, and it is not comparable to snapshot-restore figures, which measure the time to revive an already-warm process. Both are real, they measure different things, and mixing them is how people end up confused about serverless cold starts. 

delayTime also captures two costs that have nothing to do with your configuration: time spent waiting for GPU capacity in your datacenter, and image pull on a host that has never seen your image. Only three of the five phases in that interval are things the config below can move, and the other two are high-variance. A network volume can make capacity contention more likely, because it pins your endpoint to one datacenter.

The practical effect is that individual cold starts scatter more than a fixed configuration would suggest, and a single trial can land far from the median for reasons no amount of tuning would fix. Check the throttled count in the endpoint's /health response while a trial runs. Any time spent throttled is time queued for a GPU rather than booting on one, and that trial should be discarded rather than averaged in.

‍Method. FlashBoot was left enabled, which is the default on new endpoints. Set max workers to zero and confirm the endpoint reports no workers. Set max workers back to one. No worker boots at this point, because flex workers only boot against a queued job. Then submit a single 64-token request and read delayTime from the job status. Same prompt, same request, same region, multiple trials per arm.

Leaving FlashBoot on changes how you should read these numbers, and it cuts in a useful direction. Scaling max workers to zero tears the worker down, and because FlashBoot snapshots are scoped to a specific host and image rather than to your endpoint, the next boot has nothing to restore unless it happens to land back on the same host. Every trial below is therefore the snapshot-miss path. These are not the numbers a busy endpoint sees when FlashBoot hits. They are the floor you fall back to when it does not, which is exactly the case the compile cache exists to bound.

‍Baseline arm: the configuration most people ship. Defaults for everything, including FlashBoot, and no persistent caches.

‍Optimized arm: four configuration changes, detailed below.

Results and analysis

Execution time was identical on both arms, roughly 0.7s for the test request. None of this touches serving throughput. 

• Baseline: 331s / 309s / 324s (Median: 324s)
• Optimized: 93s / 89s (Median: ~91s)
• Optimized, first-ever boot: one trial (~284s)

Treat the 3.5× ratio as directional. delayTime bundles vLLM boot time together with platform costs we cannot isolate from a sample this size.

Here is the scale of that variance. On a separate proxy setup (Qwen3-8B, single 48 GB GPU, network volume, FlashBoot off) we recorded four baseline cold starts on a configuration that never changed: 273s, 355s, 387s, and 834s. The 834s trial almost certainly hit a capacity wait or a cold image pull. A 479-second spread on fixed config means we cannot say precisely how much of the 324s baseline, or the 91s optimized median, is vLLM boot versus platform fluctuation.

That proxy run produced something more useful, though. The fastest 8B baseline was 273s against 324s for the 32B FP8 model on two H200s. Roughly a quarter of the weight volume bought about 15% less cold start. Compile and graph capture dominate this penalty, not the size of the weights, and that is why the fixes below target the compile step first.

The config, in order of impact

If you want the immediate copy-paste solution, these are the four environment variables to set on your Runpod endpoint:

1. Persist vLLM's compile cache VLLM_CACHE_ROOT=/runpod-volume/vllm_cache, or any persistent path. Saves torch.compile artifacts and CUDA graphs to a persistent volume so they aren't lost when the worker dies. Requires a network volume attached to the endpoint, which is a separate step from setting this variable. 
2. Load weights in parallel SAFETENSORS_LOAD_STRATEGY=prefetch. Reads checkpoint files into the page cache ahead of materialization instead of memory-mapping them lazily. 
3. Trim CUDA graph capture sizes COMPILATION_CONFIG={"cudagraph-capture-sizes": [1 2 4 8 16 32 64]}, with MAX_NUM_SEQS=64 set to your real concurrency ceiling. Limits the batch sizes vLLM compiles at boot, drastically reducing initialization time.
4. Skip Hugging Face hub checks HF_HUB_OFFLINE=1. Drops reachability probes and version checks when weights are already local.

These are not all the same kind of thing, and the difference determines how they fail. VLLM_CACHE_ROOT is read by vLLM's own code, and HF_HUB_OFFLINE by huggingface_hub, so both need only the platform to place them in the container environment, because the consuming library reads the environment itself. The other three are vLLM engine arguments, which the worker image translates into vllm serve flags by uppercasing the flag name and swapping hyphens for underscores. That makes them the only group where a slightly wrong name gets silently ignored instead of raising an error, which is why the verification section below is not optional.

There is a second distinction worth making before you start, because it causes a silent failure that looks like the ones above. Everything in this section is a container environment variable, typed into the environment variables field on your endpoint. As we'll discuss later in this post, Runpod has a separate configuration surface for the endpoint itself: max workers, active workers, FlashBoot, idle timeout, allowed host CUDA versions, and whether a network volume is attached. Those are not environment variables, they are set elsewhere in the console or through the management API, and two of the settings above depend on the network volume in particular.VLLM_CACHE_ROOT needs it because the path it points at exists only when a volume is attached. HF_HUB_OFFLINE needs it because blocking every Hub fetch is safe only once the weights, tokenizer, and config are on disk somewhere that persists.

Here is exactly how each of these works and why they matter:

1. Persist vLLM's compile cache. VLLM_CACHE_ROOT=/runpod-volume/vllm_cache (or any persistent path)

This is where vLLM writes its torch.compile artifacts and captured CUDA graphs. The default location is inside the container, which dies with the worker. By pointing this to a persistent network volume, the compile time collapses to seconds on subsequent boots. The cache key includes the model, GPU architecture, and your exact serving flags, so changing any of them costs you one recompile.

The variable belongs to vLLM but the value is a Runpod path, and that seam is where this goes wrong. /runpod-volume exists only when a network volume is attached to the endpoint, so attach the volume before you set the variable. Set the variable without it and vLLM will happily write the cache to that path inside the container, where it dies with the worker exactly as the default did. Nothing errors. You get no improvement and no failure to debug, which is the worst possible outcome for a change whose entire value is invisible until the second boot. Note also that /runpod-volume is the Serverless mount point; on a Pod the same volume appears at /workspace.

The path above hides a tradeoff. Attaching a network volume pins the endpoint to a single datacenter, which shrinks the pool of hosts you can be placed on. For a two-GPU H200 request that can mean longer queue times and a lower chance of landing on a host that already has your image cached. You are trading compile time for placement latency. For production, prefer baking the cache into the image.

One case we have not tested: several workers writing to the same cache directory on a network volume at once. If you scale past a single worker, treat the shared-cache path as unverified rather than assuming concurrent writes are safe.

You do not need a custom Docker image for this

Nothing above requires building anything on your own. Deploy Runpod's vLLM template from the Hub, attach a network volume, and set the environment variables. That is the entire implementation, and it is the path we measured.

A custom image is worth knowing about, but it is not where to start. Baking the weights and a pre-warmed compile cache into your own image puts both on the worker's local NVMe, which removes the network volume and with it the first-boot compile penalty and the single-datacenter placement constraint described above. What it costs is a build host with the target GPU, because the compile cache key includes GPU architecture and the cache cannot be warmed on a CPU-only builder. 

2. Load weights in parallel. SAFETENSORS_LOAD_STRATEGY=prefetch (Requires vLLM ≥ 0.26)

The prefetch strategy reads checkpoint files into the OS page cache before workers materialize them, which is the win you want on network or otherwise high-latency storage. The default is lazy memory-mapping. Our measurement was 32 GB from a network volume in roughly 21 seconds, against several minutes serially. Do not chase more exotic loaders such as fastsafetensors or the RunAI streamer until the compile cache above is in place.

3. Capture CUDA graphs only for batch sizes you actually serve. COMPILATION_CONFIG={"cudagraph-capture-sizes": [1 2 4 8 16 32 64]}, with MAX_NUM_SEQS=64 

Compile and capture time scale with the size list. The default list is longer than most workloads need. 

CUDA graphs are captured per padded batch size. At runtime, a batch is padded up to the next captured size ≥ its actual size, then replays that graph. This list is a quantization grid over your batch sizes, not a property of the model itself. Each entry costs compile and capture time at boot; each missing entry costs a little padding overhead at runtime.

The recipe:

• Set max_num_seqs to your real concurrency ceiling first. If you don't pass a list at all, vLLM derives it from max_num_seqs, so a realistic ceiling already shrinks the default grid.
• Choose a coarse ladder up to that ceiling. Powers of two (1, 2, 4, 8, 16, 32, 64) is the standard choice. If traffic plateaus at a specific concurrency, add that exact size.
• Verify in the logs. vLLM prints one Capturing CUDA graphs progress line per size, which tells you exactly what boot time the list is costing you. 

Be careful with step 1, because MAX_NUM_SEQS is the one setting here that changes serving behavior rather than just boot behavior. It caps how many sequences the engine will batch concurrently, so setting it below what you actually serve trades throughput for a faster boot. If you do not know your real ceiling, find it before you set this: run your normal traffic and watch the batch sizes in the engine logs, or start high and lower it deliberately. The rest of the config in this post is boot-only and safe to experiment with on a live endpoint. This one is not.

Some model-specific wrinkles. MoE models such as the DeepSeek family compile slower per graph, so trimming the list pays double. Multimodal models typically graph only the language decoder, not the vision tower. And --enforce-eager disables graphs entirely, giving you the fastest possible boot and the worst steady-state latency, which makes it useful for dev loops and nothing else. Whatever list you pick is folded into the compile-cache key along with the model, the other flags, and the GPU architecture, so changing it costs one recompile rather than a permanent penalty.

4. Skip the Hugging Face hub checks. HF_HUB_OFFLINE=1

When your weights are already local, this removes reachability probes and version checks at import time. The time saved is small. The real value is that a boot no longer depends on the Hub being reachable.  

Set it only when weights, tokenizer, and config are all genuinely local, and understand that the failure mode is total rather than gradual. This variable blocks every Hub fetch, not just the weight download, so if the tokenizer or config cannot be resolved locally the endpoint fails every request instead of merely booting slowly. That makes it the cheapest change in this post and also the one most worth testing on a non-production endpoint first.

If your weights are not local yet, use Runpod's cached models rather than downloading from the Hub on every cold boot. The scheduler places your worker on a host that already has the model, and if no cached host is available it delays the worker start until the download completes. Either way you are not billed for download time, which for a 32 GB model is a real number. Cached models land in /runpod-volume/huggingface-cache/hub/.

Confirm each setting actually landed

Because a wrong engine-argument name does not error, verifying is not optional. Each of the changes leaves a distinct trace in the worker log, so boot once after each change and look for the corresponding evidence before you trust a measurement.

Change one variable at a time. If a setting shows no trace in the log, do not proceed to the next one, because every measurement after it inherits the ambiguity. This is the difference between learning that a technique does not work and learning that you typed a variable name the platform ignored, and those two outcomes are indistinguishable from the cold start number alone.

Platform-level levers

The config above optimizes a boot that has to happen. Two Runpod features change how often it happens at all, and a third prevents a failure that gets mistaken for one.

FlashBoot. Enabled by default on new endpoints and toggleable on existing ones. Rather than fully tearing down an idle worker, FlashBoot snapshots the worker's process state so the next request can restore it instead of booting from scratch. Two constraints determine whether you benefit. Snapshots are scoped to a specific host and image rather than to your endpoint, so if the next scale-from-zero lands elsewhere there is nothing to restore and you pay the full boot. That makes the benefit proportional to request volume: a busy endpoint hits the snapshot often, a bursty low-volume endpoint frequently misses. FlashBoot also only captures state that exists in the worker process at scale-to-zero, so a handler that lazily loads the model on first request snapshots a model-free process and gains nothing. Load the model at worker boot, before runpod.serverless.start().

So FlashBoot and the compile cache do different jobs. FlashBoot makes the snapshot-hit path fast, and the compile cache bounds your worst case on every request that misses it. Optimize only the first and your tail latency stays where it started.

Active workers and idle timeouts. If you have a predictable traffic floor, holding workers at that floor removes the cold start from the request path entirely. Active workers are always-on and carry roughly a 21% discount, but they bill continuously, so they are a fixed monthly cost rather than a safety net you get for free. They pay off somewhere around 25% monthly utilization. Separately, raising idleTimeout on flex workers from the 5-second default keeps a warm worker alive between bursts and extends billed time by that amount on every scale-down. Check current rates on the pricing page before committing to either.

CUDA version selection. This one applies only if you run a custom image, so skip it if you are on the stock worker. Make sure your endpoint's allowed host CUDA versions match what the image was built for. We watched a vLLM 0.27-based custom image crash-loop on some hosts with NCCL error: unhandled cuda error at ncclCommInitRank because its base image requires a newer host CUDA than those hosts provided. Set the minimum host CUDA version on the endpoint and the scheduler will only place you on compatible hardware. The symptom, workers flapping while jobs queue, looks like a capacity problem and is not.

What this costs to build and keep

Four environment variables reads as a cheap change, and if you stop at the network volume version it is. The accounting for the production version is longer, and it is worth doing before you commit to it.

The compile cache is keyed to model, flags, and GPU architecture, so any deploy that changes a serving flag invalidates it and costs one warmup boot. That is an ongoing obligation rather than a one-time setup. Verifying that an engine-argument variable landed is a real task, because the failure is silent, which is why the verification table above exists. Baking the cache into an image needs a build host with the target GPU, which for most teams means a new step in the pipeline rather than an existing one. And measuring any of this reliably means writing a harness that scales workers, waits out the control-plane lag described below, and samples /health for throttling, using an API key with a different permission scope than your job submissions.

Weigh that against the alternative, which is holding one active worker and never having the problem. The comparison is arithmetic: take the hourly rate for your GPU configuration from the pricing page, apply the roughly 21% active-worker discount, multiply by the hours you would keep it warm, and compare against your own loaded engineering cost for the work above plus its ongoing maintenance. For a team with a predictable traffic floor and no spare engineering capacity, the warm worker often wins.

The case for doing the config work is narrower than "it makes cold starts faster." It is that a warm worker only removes the cold start for traffic your floor covers. Above that floor you are scaling into fresh workers regardless, and their boot time sets your worst case. Teams whose traffic is spiky rather than sustained pay that worst case often and cannot buy their way out of it with a single warm worker.

How to verify on your own endpoint

1. Set max workers to 0 and confirm the endpoint reports no workers. Idle workers sit in an exited state and still count, so wait for the list to be genuinely empty.
2. Set max workers back to 1. Nothing boots yet, because flex workers only start against a queued job.
3. Submit one small job to /run and note the returned job ID.
4. Poll for the result and read delayTime, in milliseconds. That is your end-to-end cold start.
5. Compare times before and after each change, one at a time.

curl -H "Authorization: Bearer $RUNPOD_API_KEY" \
  https://api.runpod.ai/v2/$ENDPOINT_ID/status/$JOB_ID

Two things will bite you if you script this, so pay attention.

Setting max workers to 0 does not merely stop workers from starting, it pauses the endpoint, and /run then returns 409 ENDPOINT_PAUSED instead of queueing the job to wait for capacity. If you expect a queue to absorb the request, this reads as a broken endpoint.

Scaling back up does not land on both APIs at once. The management API reports workersMax: 1 while the job gateway still rejects submissions as paused, and we measured 20 to 25 seconds of lag between them. Reading the endpoint config back tells you nothing useful, so if you are automating this, retry the submit until the gateway accepts it rather than trusting the config read. That wait is not part of your cold start, since delayTime starts from the accepted job, but it will stall a script that assumes one confirmation covers both planes.

Run enough trials to see the spread and sample /health for throttled on each poll so you can discard the trials that were queuing for a GPU rather than booting. Then change one thing at a time and compare, using the log traces in the verification table above to confirm each change landed before you attribute anything to it. Those log timings are worth more than delayTime for per-lever attribution, because they come from inside the container and cannot be contaminated by capacity wait or image pull.

The fine print

These numbers are one model (32B FP8), one GPU configuration (H200×2), and one region. Absolute times will shift with weight size and tensor-parallel layout. The structure of the costs, serial load and compile-from-scratch, is model-agnostic, and that structure is what the fixes target.

The compile cache is keyed to model, flags, and GPU architecture. Budget one warmup boot per combination, or trigger it deliberately right after deploys.

Image pull time, roughly 30 to 90 seconds on a host that has never seen your image, is included in our medians. It is also a reason baked-in-image weights can beat a network volume in the worst case, because the pull and the weight fetch stop being two separate serial waits.

Five minutes of cold start is visible to whoever is waiting. Ninety seconds usually is not, and on an endpoint with enough traffic for FlashBoot to hit, most requests never pay either penalty. The four changes above are worth making because they set your worst case, which is the number your users actually notice.

To go deeper: 

Endpoint configuration docs

Cached models

Benchmarking your worker

Serverless troubleshooting

The worker-vllm repo