Speculative decoding is an inference technique that makes a large language model generate tokens faster by having a small, cheap model guess several tokens ahead, then having the large model check all of those guesses in a single forward pass. Accepted guesses are kept, rejected ones are discarded and regenerated. With the standard verification method the output is identical to what the large model would have produced on its own.
The gain is real but conditional. This guide covers the mechanism, the two things that decide whether you actually get a speedup, when the technique does nothing for you, and the exact settings that turn it on in a Runpod Serverless vLLM endpoint.
Why token generation is slow in the first place
An autoregressive model produces one token per forward pass. To produce the next token it has to read every parameter out of GPU memory again. At small batch sizes almost all of that time is spent moving weights, not doing arithmetic, so the GPU sits well below its compute ceiling while the memory bus runs flat out.
That is the opening speculative decoding exploits. If the weights have to be read anyway, a forward pass can verify several candidate tokens for close to the price of verifying one. The technique does not make the model faster. It makes each expensive memory read do more work.
How draft and verify works
The setup has two models. A target model is the one whose output you want. A draft is something much cheaper that proposes tokens.
Each step runs like this:
- The draft proposes k candidate tokens, where k is typically 3 to 5.
- The target model runs one forward pass over the whole candidate sequence and scores every position at once.
- A verification rule walks the candidates left to right and accepts them while they agree with what the target model would have sampled.
- The first rejected token is resampled from the target model, and everything after it is thrown away.
A step that accepts all k candidates advances k+1 tokens for roughly the cost of one normal step. A step that rejects immediately advances one token and has wasted the draft’s work. Real traffic sits between those, which is why the average acceptance rate is the number that matters.
Is the output identical to running the model normally?
With rejection sampling, which is the default, yes. The acceptance rule is constructed so that the resulting token distribution matches the target model’s own distribution exactly. You are not trading quality for speed, you are trading VRAM and some wasted compute for speed.
The exception is typical acceptance sampling, a looser rule that accepts any candidate the target model considers plausible rather than requiring distributional equivalence. It raises the acceptance rate and therefore the speedup, and it does change the output distribution. Use it when throughput matters more than exact reproducibility, and evaluate the effect on your own task before shipping it.
What actually determines your speedup
Two numbers, pulling against each other.
Acceptance rate is the share of proposed tokens the target model keeps. It rises when the draft is well matched to the target, and when the text is predictable. Boilerplate, code, structured output and long quoted passages accept well. Open-ended creative text accepts poorly.
Draft cost is what you pay per step for the proposals. A larger draft model guesses better and costs more. A draft that is too big erases its own benefit, which is why draft models are usually one to two orders of magnitude smaller than the target.
The practical consequence: speculative decoding is worth measuring rather than assuming. Set it up, look at the acceptance rate your traffic produces, and tune k from there. A configuration that helps one workload can be neutral or slightly negative on another.
The four ways to get draft tokens
vLLM supports several speculation methods, and they differ mainly in where the guesses come from and whether you have to host a second model.
| Method | Where guesses come from | Second model needed? | Best for |
|---|---|---|---|
| Draft model | A small model from the same family as the target | Yes | General text, when a matching small model exists |
| N-gram lookup | Repeated spans already in the prompt and output | No | Summarization, RAG, editing, anything that quotes its input |
| EAGLE and MLP heads | A lightweight head trained on the target’s own hidden states | A head, not a full model | High acceptance without hosting a second model |
| MTP and suffix decoding | Multi-token prediction built into the model, or a suffix automaton over prior text | No | Models shipped with MTP support, and repetitive traffic |
N-gram is the one most teams should try first. It needs no second model, no extra VRAM for weights, and it performs unreasonably well on any workload where the output repeats the input. If you are summarizing documents or answering over retrieved context, a meaningful share of your output tokens already appear in the prompt.
When speculative decoding will not help you
Three cases, and it is worth checking them before you spend a day on configuration.
You are running at high batch size. The whole premise is that decode is memory-bound. Once enough concurrent requests are batched together, the GPU becomes compute-bound and there is no spare capacity for verifying speculative tokens. The technique can go from helpful to actively slower. This is why vLLM exposes a setting that disables speculation above a batch size threshold, and why serving a high-traffic endpoint changes the calculation entirely.
Your bottleneck is prefill, not decode. If your prompts are long and your outputs are short, most of your latency is in processing the input. Speculative decoding does nothing for prefill. Chunked prefill and prefix caching are the levers there.
You do not have VRAM to spare. A draft model has to live in memory alongside the target, and the KV cache for speculative tokens takes room too. If you are already close to the ceiling, adding a draft model means fewer concurrent sequences, and you may lose more throughput than latency you gain.
Turning it on in a Runpod Serverless vLLM endpoint
The Runpod vLLM worker exposes speculative decoding through environment variables, so you can configure it from endpoint settings without rebuilding a container image.
| Variable | What it does |
|---|---|
SPECULATIVE_MODEL | The draft model to propose tokens |
NUM_SPECULATIVE_TOKENS | How many tokens to propose per step, the k above |
NGRAM_PROMPT_LOOKUP_MIN and NGRAM_PROMPT_LOOKUP_MAX | Window sizes for n-gram speculation, which needs no draft model |
SPECULATIVE_DISABLE_BY_BATCH_SIZE | Switches speculation off once queued requests exceed this number |
SPEC_DECODING_ACCEPTANCE_METHOD | rejection_sampler for identical output, typical_acceptance_sampler for more speed |
SPECULATIVE_DRAFT_TENSOR_PARALLEL_SIZE | How many GPUs to shard the draft model across |
SPECULATIVE_MAX_MODEL_LEN | Maximum sequence length the draft model supports |
TYPICAL_ACCEPTANCE_SAMPLER_POSTERIOR_THRESHOLD | How plausible a token must be to be accepted, if using typical acceptance |
TYPICAL_ACCEPTANCE_SAMPLER_POSTERIOR_ALPHA | Scaling factor on the entropy-based acceptance threshold |
The full list is in the vLLM environment variables reference. A reasonable first configuration is n-gram speculation with NGRAM_PROMPT_LOOKUP_MIN at 2, NGRAM_PROMPT_LOOKUP_MAX around 4, and NUM_SPECULATIVE_TOKENS at 3, because it costs no extra weights and tells you quickly whether your traffic is predictable enough to benefit.
If you need a speculation method the worker does not expose, build your own container against a current vLLM release and deploy that instead. The worker is a convenience layer, not a ceiling.
Sizing the GPU
Budget for the target model, the draft model, and KV cache for both. A useful way to think about it: the draft is small enough that its weights rarely decide your GPU tier, but it does eat into the headroom you were using for concurrency.
Prototype on a pod before you deploy to Serverless. You get a shell, you can watch acceptance rate directly, and you can change k without redeploying an endpoint. Secure Cloud rates:
| GPU | VRAM | Secure Cloud | Fits |
|---|---|---|---|
| RTX 4090 | 24 GB | {{gpu:rtx-4090}}/hr | A quantized 8B target with a small draft, tight on concurrency |
| L40S | 48 GB | {{gpu:l40s}}/hr | An 8B target at BF16 with a 1B draft and room to batch |
| A100 PCIe | 80 GB | {{gpu:a100-pcie}}/hr | Larger targets, or comfortable concurrency on a mid-size one |
| H100 PCIe | 80 GB | {{gpu:h100-pcie}}/hr | Same capacity, higher memory bandwidth, which is what decode is limited by |
Bandwidth is worth paying attention to here specifically. Speculative decoding attacks a memory bandwidth bottleneck, and a card with more bandwidth is already less constrained by the thing you are trying to fix. The two improvements overlap rather than stacking cleanly.
The managed alternative
If the tuning loop is not where you want to spend engineering time, Runpod Overdrive is an inference optimization engine for open-source LLMs on Serverless. It composes a set of techniques against your specific model and traffic profile rather than a benchmark configuration, which is the same problem this guide describes solving by hand: a generic default leaves capability on the table, and the right configuration depends on workload behavior you have to measure.
Frequently asked questions
Does speculative decoding change the model’s output?
Not with rejection sampling, which is the default. The verification rule is designed so that the token distribution matches what the target model would have produced alone. Typical acceptance sampling is the exception: it accepts a wider set of candidates for more speed and does change the distribution.
How much faster is speculative decoding?
It depends entirely on your acceptance rate and your batch size, so any single number is misleading. Predictable output at low concurrency benefits most. High-concurrency serving may see no benefit at all, because the GPU is compute-bound rather than memory-bound at that point. Measure it on your own traffic rather than trusting a benchmark figure.
Do I need a second model to use speculative decoding?
No. N-gram speculation draws its candidates from text already in the prompt and generated output, so there are no extra weights to host. It works well on summarization, RAG and editing workloads, where output frequently repeats the input. Draft models and EAGLE-style heads generally accept better, at the cost of extra memory.
What is a good number of speculative tokens?
Three to five is the usual starting range. Proposing more raises the payoff of a fully accepted step but wastes more work when a rejection comes early, so the best value tracks your acceptance rate. Start at 3, measure, and adjust.
Can I use speculative decoding with quantized models?
Yes, and the combination is common, since both target the same bottleneck from different directions. Quantization shrinks the weights that have to be read each pass; speculation gets more tokens out of each read. Confirm the draft and target are compatible in your serving stack before assuming they compose cleanly.
Why did speculative decoding make my endpoint slower?
Almost always batch size or acceptance rate. Under heavy concurrency the GPU is compute-bound and verification competes with real work, so SPECULATIVE_DISABLE_BY_BATCH_SIZE exists precisely to switch speculation off above a threshold. A poorly matched draft model produces the same result at any batch size, by paying for proposals that get rejected.
Getting started
Deploy a vLLM endpoint, start with n-gram speculation because it costs nothing to try, and read the acceptance rate before tuning anything else. If your traffic accepts well, move to a draft model or an EAGLE-style head and measure again. If it does not, the honest answer is that speculative decoding is not your bottleneck, and prefix caching or a different GPU tier will do more for you.
