Scale should make the model cheaper

‍Editor’s note: This is the third of three posts on the same decision: whether to rent a model by the token or run your own weights. The first was about control, or who gets to deprecate or reprice the model you shipped. The second was about customizability, and how a model trained on your data beats a general one. This one is about cost.

Ask most teams building on AI what they will spend on tokens next quarter and you will get a range, not a number. It’s a range even a company as data-driven as Uber couldn’t forecast accurately: four months into the year the company had spent its full annual AI budget, and in June it capped every employee at $1,500 a month per AI coding tool.

By renting I mean paying per token, which covers the closed frontier models and the open-weight ones served through somebody else's API. Either way, serving your millionth user costs what serving your first one did. Doing more of it never makes it cheaper. You can negotiate a better rate, but it's still a rate, and it still applies to every request you add. 

It’s showing up in the margins, too. In May, ICONIQ surveyed roughly 305 executives at software companies building AI products. Average gross margin came in at 45% for 2025, against the 80% or so that defined software businesses for the last twenty years. The line in the report I keep coming back to is this: as products scale, talent's share of cost falls while inference’s share rises.

Every other line in a software P&L improves with volume. Headcount per dollar of revenue falls. Hosting gets renegotiated. Support gets automated. The model bill moves the other way, because it's priced per unit of the exact thing your product does more of when it works. 

The volume side of that is worse than it looks. In April, researchers led by the University of Michigan, with co-authors at Stanford, Microsoft AI, and Google DeepMind, measured what agents actually spend across eight frontier models on SWE-bench Verified, a benchmark built from real GitHub issues. Agentic work ran on the order of a thousand times the tokens of a chat request. Two runs of the same task differed by as much as 30x. And when they asked the models to predict their own token usage, the models underestimated it. So the fastest-growing part of your bill is the part you can't forecast.

Where the levers are

Almost everything that permanently lowers your cost per request requires the weights. 

Quantize the model, which drops the precision of its numbers, and it runs on cheaper hardware. Distill it, which trains a smaller model to imitate the big one, and a smaller model does the same job for less. Put your own small model in the cheap slot and save the frontier for the hard 10%. Batching and caching are available to renters too, at published discounts, so the difference there isn't access. It's that you set the batching policy and decide what stays resident, rather than taking a vendor's defaults and TTLs. Each of those is a change to your unit economics that you own from then on.

Rent instead and your levers are the prompt and the volume, which means you can write shorter prompts or you can serve fewer customers. The rate can also change. When DeepSeek raised its API prices on August 16 by anywhere from 50% to more than 1,100%, depending on the model, token type, and the hour, anyone already running those weights on their own hardware paid the same as the day before. 

Owning turns the bill into a number you set

The cost of compute is hours times the rate. Take the hourly rate of Pods on whatever GPU the model needs, multiply by the hours you plan to run it, and you have next month's number before next month starts. It holds while your request volume grows into it. A fixed footprint spread across more work gets cheaper per unit of work, which is how software economics is supposed to behave.

It isn't free. Somebody has to serve the model, monitor it, get paged when it breaks, and that's engineering time you'd otherwise spend on the product.

The strongest case against owning

I can think of two good reasons: 

The first is that per-token prices keep falling. Stanford's AI Index tracked the cost of running a model at GPT-3.5's level of capability dropping more than 280x in two years, from roughly $20 per million tokens at the end of 2022 to roughly $0.07 at the end of 2024. And two-thirds of the companies in that ICONIQ survey improved their per-query economics without owning a model at all, through better routing, inference management, and revenue growth outrunning the bill.

The second argument is utilization, and this is where self-hosting usually goes wrong. A GPU running 10% of the time makes every hour of real work cost ten times the rate you're paying. Most products have spiky traffic, which a rented endpoint handles well and a reserved GPU handles badly.

Both arguments hold at low or unpredictable volume, where renting is the right call. The turn comes when your volume is steady enough that a fixed monthly cost beats a per-token bill or when your traffic is spiky and you can scale to zero between requests. That is the problem Runpod Serverless exists to solve.

Where I'm unsure

That same ICONIQ report projects gross margins expanding to 53% this year and 59% next. If that holds, the compression is a phase and renting works out fine for most people. I don't know whether it holds. The projection needs the model bill to cooperate, but I wouldn’t bet a business on it. 

Software has been a good business because the cost of serving each additional user keeps falling. A per-token bill changes that math. Owning the weights doesn't restore it, but it does make the cost of a request something your engineers can lower, which is work you control. 

The cost I didn't put a number on here is the engineering time to run the thing. That's what I'll write about next: what it takes to run custom models in production once they're yours. If you've run this math on your own workload and it came out the other way, send me the numbers.