Private GPU pools and reserved GPU capacity, explained
When a private GPU pool beats on-demand GPUs, how reserved capacity is billed, what happens when traffic bursts above the pool, and what usage data to bring before you commit.
All
Switching the serving engine moved embedding throughput by up to 11x in our benchmarks, on the same GPU, running the same model on the same workload. No card upgrade on our price list buys you 11x. This post set out to find the GPU with the best throughput per dollar for embedding models, and the answer is below, but the engine question comes first: which one are you running, and does it like your model?
This started with a user we could not give a number to. They embed academic text with the BAAI bge-m3 embedding model, 568M parameters at FP16, producing 1024-dimensional dense vectors from 512-token chunks. They asked what throughput to expect on a variety of different GPUs, because they wanted to know whether their rented hardware delivered full value. Our internal benchmark suite covered chat, image, video and speech models, but not one embedding model, so we fixed that.
What we measured
We picked seven embedding models from what users actually serve on Runpod: bge-m3, bge-large-en-v1.5, MiniLM-L6-v2, multilingual-e5-large, nomic-embed-text-v1.5, Qwen3-Embedding-0.6B and Qwen3-Embedding-8B. bge-m3 headlines the set because our users serve it more than any other embedding model.
Each model ran on up to 24 GPU types at three input lengths: short queries around 50 tokens, passages around 512 tokens and long documents around 2,000 tokens. We measured two modes for each combination:
- Bulk mode saturates the card with large batches and measures maximum throughput in texts and input tokens per second. This is the mode that matters when you need to embed 10 million documents tonight.
- Online mode walks a concurrency ladder of 1, 8 and 32 simultaneous requests and measures end-to-end latency at p50, p95 and p99, which is what a user waiting on a search box experiences.
We ran three serving engines, because users do not all serve embeddings the same way:
- vLLM on pods as the cross-GPU baseline, because vLLM embedding support covers all seven models and the widest set of cards. Most vLLM cells ran on a pinned May 2026 build, with a minority on the newer 0.28 release.
- Text Embeddings Inference (TEI) 1.9 on pods, because that is what most pod users run.
- The stock Infinity worker (1.1.4) on Serverless.
We measured every number in this post from the client side, on rented Secure Cloud pods and live Serverless endpoints. The completed runs produced roughly 1,940 result rows, with about 10 failed requests out of roughly 600,000. Each cell is a single run on one host of that GPU type; we did not repeat cells across multiple hosts, so an individual cell can carry host-to-host variance that a multi-host median would smooth out. To compute cost, we divide the card's Secure Cloud hourly list price as of September 10, 2026 by its measured token throughput and express the result as dollars per billion input tokens. A billion tokens covers roughly 2 million typical 512-token chunks. Every table names the engine that produced its numbers, because the engine turned out to be the largest variable we measured.
In this post, we feature a targeted excerpt of that dataset that focuses on the serving engine comparison and bge-m3 performance with 512-token passages across six key GPUs. Accompanying this post is the complete benchmark table, detailing all evaluated graphics cards, three sequence lengths, engine labels for every row, and tail latencies across all concurrency levels. If your specific GPU model or sequence length isn't displayed in the charts below, refer to the full dataset before making performance assumptions.
The benchmark compares serving engines with each other. It does not measure raw sentence-transformers in a Python service, which is the stack many teams start on. If that describes your baseline, run a one-hour trial of TEI or vLLM on your current card before you read any chart here as a hardware recommendation, because the engine migration is the first variable to test.
Pick your engine before you pick your GPU
Change nothing but the engine and throughput can swing anywhere from a rounding error to roughly 10x, depending on the model and the card.
Two rows need a footnote. The e5-large median hides a skew: most cards sit at 1.0 to 1.3x and a few outliers stretch to 12x. And Qwen3-Embedding-8B's double-digit vLLM wins appear on the A100, H100 NVL, RTX A6000 and RTX 6000 Ada; on the A5000 and the consumer 4090 and 3090, the two engines land near parity.
TEI 1.9 carries years of hand-tuned kernels for BERT-era encoders, and MiniLM, e5 and bge-large all belong to that family. Its Qwen3 support shipped more recently, and the measured margins above reflect that. vLLM mirrors the same history: it leads on the newer LLM-style embedding models and trails on the older encoders.
If you serve a BERT-family model, run TEI. If you serve a Qwen3-style embedding model, run vLLM. If you serve bge-m3, the engines tie at a median ratio of 1.08 across 48 matched cells, so pick on operational preference. Get this wrong and you typically leave 1.5 to 4x of throughput on the table, and up to 10x or more on the wrong card, before you have chosen a GPU at all.
Two coverage gaps to know about. TEI has no sm_120 build, so the RTX Pro 6000, 5090, Pro 4500 and Pro 4000 are vLLM-only in our data. And TEI 1.9 cannot load nomic-embed-text-v1.5 because of a config.json defect, which is why nomic is absent from the engine table despite being BERT-family.
Two checks belong in any engine migration. First, confirm both engines return equivalent vectors for your model, because pooling and normalization defaults differ across engines, and a throughput win means nothing if your recall shifts. Second, if you rely on bge-m3's sparse or multi-vector outputs rather than its dense vectors, confirm your engine serves those output modes before you commit.
We also checked whether our chat-model benchmarks predict the embedding results, and the answer splits. Raw speed ranks almost identically: across 22 GPUs, the rank correlation between Qwen2.5-7B peak chat throughput and bge-m3 passage-bulk throughput is 0.99. Cost at saturation does not. Chat serving compresses the cost gap between big and small cards: an H100 SXM, an RTX Pro 6000 and a 4090 all land near $0.05 per million output tokens, tied for second behind the A5000 at $0.03, so you buy the big card for headroom, model size and latency rather than economics. Embeddings never close that gap: the H100 SXM costs $1.53 per billion input tokens against the A5000's $0.78, the H200 $1.85 and the B200 $2.75, and every embedding model in this set fits on a card that rents for {{gpu:rtx-a5000}}/hr on Secure Cloud, so there is nothing for the flagship's extra memory to buy. Embeddings belong on small GPUs; chat earns big ones.
On pods, single-query latency is a solved problem
For bge-m3-class models at query and passage lengths, single-query p50 latency came in between 3 and 16 milliseconds on TEI across the cards we tested, and no user waiting on a search box notices the difference between 7ms and 14ms. The ceiling rises outside that scope: vLLM reaches 33 milliseconds on some cards, 2,000-token documents run 8 to 59 milliseconds, and Qwen3-Embedding-8B on small cards runs 70 to 480 milliseconds. TEI led vLLM in roughly nine of ten matched pairs, with a median advantage of about 1.4x overall and 1.6x at query length, and it rarely mattered. On pods, pick your GPU on throughput per dollar and ignore latency at the median. We measured the full concurrency ladder at p50, p95 and p99; if you carry a strict p99 budget under concurrent load, read your card's row in the full table rather than extrapolating from single-query medians.
The value chart: bge-m3 at 512-token passages
We measured these TEI numbers on the workload the user asked about: bge-m3 embedding 512-token passages.
The price column shows the live Secure Cloud rate. The cost column was computed from September 10, 2026 list prices, so if the two disagree, divide the live price by the measured throughput yourself.
Read the spread before the ranking. The worst value here costs 2.7x the best, and the entire range stays in single-digit dollars per billion tokens, which means embedding a serious corpus costs almost nothing on any of these cards. The RTX A5000, which rents for {{gpu:rtx-a5000}}/hr on Secure Cloud, leads on value on both engines. The H200 takes the throughput crown at roughly 1,345 texts per second, and you pay for that speed in dollars per token; it wins when wall-clock time matters more than cost, such as a re-index with a deadline.
Buying the most expensive GPU does not buy safety here the way it can for training. Embedding inference runs a single forward pass, and over-buying compute for that shape burns budget without buying throughput.
These figures price a card you keep busy. If your traffic is spiky and user-facing and the card idles most of the day, dollars per billion tokens stops being the right unit, and the platform choice in the Serverless section below matters more than the card choice.
For small models the floor drops further. MiniLM-L6-v2 on TEI sustained roughly 650,000 tokens per second on an RTX 2000 Ada ({{gpu:rtx-2000-ada}}/hr, Secure Cloud), which worked out to $0.10 per billion tokens at September 10, 2026 prices.
The comparison that flips with the engine
The user who prompted this benchmark asked about the A40, the RTX 6000 Ada and the L40S. On vLLM, the L40S beats the RTX 6000 Ada by roughly 47%, a clear win. On TEI, which is what that user actually runs, the two cards tie at roughly 275 texts per second, and the RTX 6000 Ada costs less per token at $1.65 against $2.14.
The A40 has one measured data point so far: on vLLM it does roughly 238 texts per second, which worked out to about $1.19 per billion tokens at its September 10, 2026 list price; the current Secure Cloud rate is {{gpu:a40}}/hr. Our TEI runs on the A40 have not landed yet because of card availability; we will update this post when they do rather than substitute a spec-sheet estimate, because nothing on these cards' spec sheets predicts the tie above.
Serverless is a platform choice, not a GPU choice
Serverless answers a different question, so keep it off the pod charts. At low concurrency, the request path, not the GPU, dominates Serverless latency. On the Ada 80 tier, single-query p50 runs 650 to 900 milliseconds regardless of payload size, against 6 to 14 milliseconds pod-local. Some tiers also show a second mode where p50 pins near 5 seconds, which we have not root-caused yet, so treat single-request Serverless latency as a platform behavior under investigation rather than a GPU property. A chart that puts pods and Serverless side by side would misattribute all of this to hardware.
Give an endpoint real work and the GPU shows up: bge-m3 passage embedding sustained 33,000 to 42,000 input tokens per second in bulk mode on the Ada tiers, and 7,000 to 12,000 tokens per second at 32 concurrent online requests across the four tiers we tested.
Run pods when users wait on individual queries or when a steady bulk job keeps a card saturated. Run Serverless when traffic spikes and scale-to-zero economics dominate, and design for concurrency rather than single-request latency. Two compatibility notes on the stock Infinity worker (1.1.4): it does not boot on the five Blackwell-generation Serverless tiers, and it does not serve Qwen3-architecture embedding models, so bring a vLLM-based worker image for either case.
How to choose
- Match the engine to the model family: TEI for BERT-family encoders, vLLM for Qwen3-style embedders, either for bge-m3.
- For bursty or low-volume work, rent the cheapest card that fits your model in VRAM, which on our measurements is the RTX A5000 for bge-m3-class models.
- For steady bulk embedding, pick the best dollars per billion tokens at your real batch size and input length, and measure at your own sequence length if it differs much from 512 tokens.
- For user-facing queries, run pods; bge-m3-class models are already fast enough at the median on every card we tested.
Start with the engine table, then the value chart, and rent the RTX A5000 before you rent anything bigger. If it saturates and the job still is not fast enough, the chart tells you exactly what each additional dollar buys.
Get Started
Runpod Serverless runs your container as an autoscaling endpoint that scales to zero and bills by the second.
- Try Serverless: Runpod console
- Python SDK: github.com/runpod/runpod-python
- Examples: github.com/runpod-workers
- Docs: docs.runpod.io/serverless/overview
- Discord: join us here
