Clear models, fast starts: building Runpod's model store

We built Model Store and the start of a private Model Repository because we got tired of re-downloading the same multi-gigabyte weights from Hugging Face or some S3 bucket every single cold start. If you're running serverless GPU inference at any scale, this cost compounds fast. Now there's a tiered cache: host-local disk first, a datacenter-scoped network volume second, remote origin only when nothing closer has the bytes.

The other piece is explicit model version pinning, so rollouts and rollbacks aren't a guessing game, plus scheduling that actually knows where a model's bytes physically exist before it decides where to run a worker.

If you're serving large artifacts at scale, most of your wins come down to three questions: where do the bytes live, who's responsible for moving them, and when is it actually safe to schedule a worker.

The problem with downloading on every cold start

Model weights are large, deployments are frequent, and traffic spikes without warning. If your only move on a cold start is "download from origin," then you've got three problems stacked on top of each other.

First, your cold-start latency is almost completely transfer time. Your code isn't slow, the network is. Second, your reliability is now whatever Hugging Face's uptime is that day, which you don't control. Third, every host that schedules a worker for that model downloads the exact same bytes independently, and that's a lot of wasted bandwidth.

And there's a question that's more challenging than it should be to answer: which model version is actually serving traffic right now? If workers are pulling "latest" or whatever happened to get cached, good luck answering that with confidence during an incident.

We wanted the deploy flow to stay simple, serving to get more predictable, and the scheduler to use the fact that model artifacts might already be sitting somewhere close by. 

This blog post explains how and why we built our Model Store.

Our approach

We treated model serving as an artifact distribution problem and this guided three major design choices:

1. Tiered caching, without touching the developer workflow. 
   
   1. Warm path: the model's already cached close to compute, use it. 
   2. Cold path: if nothing local exists, then fetch from origin. 
   3. Fallback: if origin is slow or down, the system keeps serving regardless.
2. Locality-aware placement. When scaling out, prefer hosts that already have the model, or can reach it from a nearby cache, over ones that don't. This is a real scoring change in the scheduler: host matching now factors in whether a model's on the host, on a network volume in that datacenter, or not nearby at all.
3. Explicit "ready" gating. A worker doesn't start until its model is verified and available. If a shared cache needs to get populated first, that happens before more workers get scheduled, not at the same time. That ordering is what keeps a scale-out burst from turning into a redundant-download storm.

Technical deep dive

Two lifecycles, easy to conflate

There's a serving lifecycle: a worker starts, loads a model from a known path, and starts. And there's an artifact lifecycle: files get fetched, verified, cached, eventually evicted. Mix these up and you'll spend a lot of time debugging the wrong layer.

Making the artifact lifecycle explicit is what lets the serving lifecycle be boring. A worker that already knows exactly where its bytes are, and that they're verified, has nothing interesting to do at startup. That's the whole point.

Architecture and network volume caching

Three projects are involved:

1. runpod is the control plane. Scheduling, orchestration, and the database tracking model-to-host and model-to-network-volume assignments live here. This is also where locality-aware scoring lives, plus the state machine deciding whether a model needs assignment, is already in transit, or is good to go.
2. host is the agent running on each GPU machine. It tracks model state locally, runs the transfer engine (downloads from origin or network volume, uploads to network volume when it's been assigned that job), and reports status upstream.
3. proxy is a passthrough. It relays assignment instructions and network volume state between the control plane and hosts with no extra coordination logic of its own.

Figure 1. Architectural overview of multi-component communication. 

When we first started, every scale-out re-downloaded from the origin, independently, on every host. This  improvement assigns a model to a network volume once per datacenter, then defers scheduling additional workers until that download finishes. One download, then everyone in that datacenter reads the shared copy.

Scaling an endpoint with a model: the actual sequence

1. User deploys an endpoint and specifies a Hugging Face model ID in the endpoint configuration.
2. Scheduler checks two things: is the model on a specific host already, and is it cached on a network volume in the target datacenter.
3. If it is not on the network volume yet, and the volume has capacity, then one host gets assigned to pull it down. Any other workers that would scale out right away get deferred.
4. That host downloads from its assigned source, then uploads to the network volume async, once its own local copy is deployed.
5. Status flows host → proxy → control plane as the transfer runs. Once every blob is hash-verified, the assignment flips to complete, and every worker after that reads from the network volume instead of hitting the origin again.

If you're debugging a slow scale-out: a host that stops reporting progress gets treated as stalled after about five minutes of silence. It doesn't hang forever waiting.

What keeps this correct under failure

Four things actually do the reliability work here, not vibes, but actual mechanisms.

1. Content verification: hash is the canonical ID for a model version. Mismatch fails the transfer outright. We're not serving bad bytes to a worker because nobody checked.
2. Transfers retry by design. Every transfer moves through defined states (initialized, pending, transferring, verifying, completed, failed, canceled), so a retry is just a normal transition, not a special case someone bolted on later.
3. Fallbacks are explicit. Is the network volume down? Skip it, mark that assignment failed, fall back to origin. We don't block a request on a layer that isn't working.
4. Disk safety is the part we're not done with yet. Network volumes run on existing MooseFS infrastructure scoped per datacenter, and quota and eviction policy are still open problems, not solved ones. In the current implementation, if a network volume reaches capacity, the system skips the NV tier and falls back to origin downloads for that model. Workers still start, just without the caching benefit. More on that later, in What's Next.

Key technical decisions

Optimize for a friction-free loop over a public catalogue, early. We aligned on utility over discovery: import, fine-tune, version, and one-click deploy mattered more than a browsable catalogue. A catalogue solves a discovery problem. We had a workflow problem first, so that's what we fixed first.

Explicit model versioning and version pinning. Pin model@version instead of trusting "latest," and "what's actually serving" stops being a mystery. Rollbacks become something you do on purpose, not something you cross your fingers about. This matters most during incidents: if you can't answer "which exact version is running" in under thirty seconds, you're debugging blind.

Defer scheduling until artifacts are ready. This sounds obvious until you've actually watched the alternative: a scale-out burst hits hosts that don't have the model yet, and now you've got a stampede of redundant downloads and workers coming up at wildly different times. Deferring kills that problem outright.

Datacenter-scoped shared cache via network volumes. One download benefits every host in that datacenter, not just the first one that happened to need it.

We looked at two alternatives and passed on both. Always pulling from Hugging Face or S3 is simplest, but it's slow at scale and you inherit every outage upstream. Pre-baking models into container images gets you a faster startup sometimes, but the images get huge, iteration slows down, and versioning turns into a mess. We saw this firsthand with the ComfyUI worker line: pre-baked variants per model family were the old pattern, and moving to a modular base image was the better call long-term.

Key takeaways

Make where the bytes live an explicit part of your architecture. Don't let it be something you stumble into while debugging.

Treat model artifacts as a first-class system, not a side effect of container startup. Treat them like an afterthought and you get mystery latency nobody can explain.

Use orchestration to turn distributed storage into predictable runtime behavior. The storage layer itself doesn't need to be exotic. The coordination on top of it is where reliability effectively  comes from.

If we did this again, we'd build customer-facing visibility into cache tier and model version usage earlier, instead of treating it as a follow-on. And the lesson that generalizes past this one project: the more explicit your state machine, the less guesswork you're stuck with when something breaks.

Try it yourself

We've already shipped cached models on Runpod Serverless. Pick a Hugging Face model when you configure your endpoint, and Runpod tries to start workers on hosts that already have it. On a warm host, workers are ready in seconds (actual model load time into GPU memory depends on model size). On a cold host with no cached copy, you wait on the download, but you're not billed for it. Note that each endpoint currently supports one cached model at a time.

If a Hugging Face repo has multiple quantization variants, all of them are downloaded currently, not just the one you selected. Selective quantization download is on the roadmap.

For custom workers, cached models live at /runpod-volume/huggingface-cache/hub/models--{org}--{name}/snapshots/{hash}/. There's a working example at github.com/runpod-workers/model-store-cache-example: it resolves the cached snapshot path and loads the model in offline mode.

The private Model Repository described above, with version pinning and pod-to-repo offload (the ability to export model weights from a running pod directly into the repository), is currently in beta and isn't publicly documented yet. We'll link it here once it ships.

Learn more

• Cached models on Runpod Serverless
• Try Runpod Serverless: Runpod console
• Discord: join us here

‍