Blog

Cold Starts Were Never the Real Problem

Flash deploys Python functions as serverless GPU endpoints in under 30 seconds. FlashBoot cuts serverless GPU inference cold starts to under 200ms. Here's how both work.

Your model works. The inference logic is solid. You've tested it locally and it's fast. Then someone asks you to ship it.

Before Flash, that meant managing infrastructure config, Docker registries, and environment variables. If you wanted GPU access, you were also picking instance types, configuring CUDA versions, and making sure your base image matched your driver stack. You spent two to three days on setup before a single inference request hit production.

That setup friction was our starting point for Flash, our framework for deploying Python functions to serverless GPU endpoints..

Automating the build and how Flash deploys GPU endpoints

Flash deploys Python functions as serverless GPU endpoints, allowing you to write a handler, decorate it with @Endpoint, and, once your code is ready, go live in under 30 seconds. By bundling your Python dependencies into a separate artifact that is mounted into the worker at startup, Flash avoids the overhead of rebuilding and pushing full Docker images. With PyTorch and CUDA drivers pre-cached across the fleet, your artifact typically results in a diff under 50 MB, keeping load times under a second.

It supports two execution patterns. Queue-based runs jobs asynchronously through a job queue, where workers pull and process requests independently. This is the right choice for batch inference, document processing, and anything where per-request latency isn't the primary constraint. Load-balanced distributes jobs across available workers synchronously using a round-robin routing layer, which is what you want for real-time inference APIs where you need consistent low-latency responses.

Workers on both patterns autoscale between your configured minimum and maximum counts based on queue depth and incoming request rate. When traffic drops, workers exit and you stop paying for them. See Runpod Serverless pricing for how per-second billing works.

What changes under the hood is what Flash actually deploys. In the standard Docker model, your Python dependencies live inside the image. Changing a dependency means rebuilding and repushing the image, and if that image isn't cached on the machine running your worker, the full pull happens again. As of mid-2026, a PyTorch 2.x base image runs around 12 GB. A fine-tuned model baked into the image can push that to 30 GB or more. Pushing and pulling images this size can take up to 10 minutes - not exactly production-ready.

When you run flash deploy, Flash bundles your Python dependencies into a .tar.gz stored on Runpod's infrastructure and mounted into the worker at startup. The base image (PyTorch, CUDA drivers, system libraries) is pre-cached across the fleet. Your artifact contains only your code and dependencies, not the full image stack. In practice, that artifact is typically a few hundred MB, and when you update your handler or change a dependency, the diff is usually under 50 MB. The expensive layers are already on the machine, and load times fall to under a second.

The hidden cost of serverless GPU inference: cold start latency

Removing deployment friction exposed the next hurdle, cold start latency.

When an endpoint scales to zero, which it should, because you shouldn't be paying for idle GPU time, the next request waits for a worker to start. Runpod allocates a GPU from the available pool, pulls your image if it isn't cached on that machine, starts the container, runs any init scripts, and loads your model weights into VRAM before the worker is marked ready and your handler starts executing. For a 7B parameter model, loading weights from a network volume takes around 15 seconds. Downloading the same model from HuggingFace at startup can take up to two minutes. End to end, cold starts typically run 30 seconds to two minutes plus depending on image size and model.

You cannot reduce the time it takes to move weights from disk to VRAM. It is a physical constraint. We used a 7B parameter model as an example, and at 13B or 70B parameters, the load time grows proportionally. The obvious fix is to keep a warm worker running. This works, but it means paying for a GPU sitting idle between requests. You end up choosing between cold starts and idle GPU spend. Both options shift the cost to the developer.

FlashBoot cuts GPU cold starts to under 200ms

Runpod FlashBoot is the optimization engine that cuts serverless GPU cold starts to under 200ms by keeping a pool of pre-warmed workers ready before requests arrive.

While Runpod Flash is the application framework you use to write and deploy the code, FlashBoot is the low-level systems mechanism that keeps a pool of pre-warmed workers ready before requests arrive, so after usage scales down to zero you can spin back up instantly. You can enable it for your endpoint with a toggle in Runpod Console.

FlashBoot relies on container pausing rather than stopping. When a worker finishes a job and would normally exit, we freeze the process at the OS level, but the GPU memory allocation and model weights stay loaded in VRAM. A Host Daemon running on each machine subscribes to a Redis pubsub channel. When a new job arrives, the scheduler publishes a message to that channel and the daemon calls docker unpause on one of the paused containers. The container resumes from exactly the state it was paused in, skipping image pull, container initialization, and model load entirely.

FlashBoot resumes the container exactly where it was in about 200ms, while a true cold boot, lacking warm workers and cached layers, takes much longer (as in seconds or minutes compared to milliseconds).

Endpoints still scale to zero when genuinely idle. Paused containers don't hold an active GPU allocation indefinitely. If they sit idle past a timeout, the system releases them.

For predictable traffic spikes, such as a launch, a load test, or a scheduled batch job, flash dev --auto-provision provisions every @Endpoint-decorated function in your project before any requests arrive, so the first request to each endpoint isn't a cold start. Endpoints are cached and reused across server restarts.

Capacity contention and Priority FlashBoot

Standard FlashBoot works well under normal conditions, but pre-warmed containers can be evicted if GPU capacity is constrained by another workload. When that happens, the next request triggers a full cold start.

Priority FlashBoot (PFB) adds an eviction guarantee on top of the standard FlashBoot mechanism. We mark these containers as last-to-evict. If a PFB container does get evicted under sustained pressure, the daemon automatically reprovisions it proactively as soon as capacity becomes available so the container is running when you need it.

PFB is built for production inference APIs where P95 latency is a hard requirement and you're running the kind of workload where a cold start under load breaks an SLA. Because it guarantees dedicated standby capacity, it's a conversation with your Runpod account team rather than a self-serve toggle.

Our current state

Inference has structurally different traffic characteristics than training. Training jobs are long, scheduled, and predictable. Inference requests are short, user-facing, and bursty. Serverless maps well to that shape. Pay-per-request pricing, autoscaling worker pools, and no idle GPU spend between traffic peaks all fit bursty inference workloads. The problem is that the full cost of starting a worker lands on the first request after a quiet period.

The tradeoff gets worse as workloads become more compositional. An agentic pipeline that fans out twenty sequentially GPU calls doesn't absorb one cold start - it absorbs twenty. At 30 seconds each, that's ten minutes of blocking latency before the pipeline completes. Cold start elimination isn't a UX improvement, it's a hard prerequisite for the workload being viable at all.

The result of our efforts is encouraging. Cold start times improved measurably between beta and GA. The remaining gap is the true first boot, with no warm workers, no cached layers, and no FlashBoot state, which is where work on lazy container loading and on-disk image pre-caching is headed.

The artifact-based deployment model (FlashBoot) is already moving the workload mix in the right direction, and the developer loop (Flash) is meaningfully streamlined from what it was a year ago.

Try Flash and FlashBoot

Flash requires Python 3.10, 3.11, 3.12, or 3.13 and runs natively on macOS and Linux. Windows users can run Flash through WSL2.

Install Flash:

pip install runpod-flash

Authenticate with your Runpod account:

flash login

This saves your API key security and allows you to use the Flash CLI to run @Endpoint functions. Note that the above will open a browser for OAuth. If your environment doesn't support a browser, then you will need to set RUNPOD_API_KEY as an environment variable instead.

This example deploys a single Python function as a serverless GPU endpoint that runs your inference logic on Runpod.

from runpod_flash import Endpoint, GpuGroup

@Endpoint(	
   name="inference",	
   gpu=GpuGroup.ANY,	
   dependencies=["torch"]
   )
async def infer(data: dict) -> dict: 
   import torch    
   # your inference logic here    
   return {"output": None}

To enable FlashBoot on an existing endpoint, go to your endpoint settings in the Runpod console and toggle FlashBoot on. There's no additional cost. Priority FlashBoot is available for production workloads that need guaranteed warm capacity under contention. Reach out to your Runpod account team to get it configured for your endpoint.

This example splits a workload across two endpoints, running lightweight preprocessing on a CPU worker and model inference on a GPU worker, with the orchestration handled locally. Please note that CPU endpoints are currently only available in our EU-RO-1 region.

import asyncio
from runpod_flash import Endpoint, GpuType, CpuInstanceType

# Runs on a CPU worker - lighweight preprocessing 
@Endpoint(
   name="preprocess",   
   cpu=CpuInstanceType.CPU5C_4_8,   
   dependencies=[]
)
def preprocess(raw: bytes) -> dict:
     # your preprocessing logic here
     return {"cleaned": None}

# Runs on a GPU worker - model inference
@Endpoint(   
   name="infer",
   gpu=GpuType.NVIDIA_A100_80GB_PCIe,
   dependencies=["torch"]
)
async def infer(data: dict) -> dict:    
   import torch    
   # your inference logic here        
   return {"output": None}

#Orchestration runs locally and calls each endpoint in sequence
async def main():   
   cleaned = await preprocess(b"your raw input")
   result = await infer(cleaned)   
   print(result)

asyncio.run(main())