Autoregressive language models normally generate one token at a time. Each token requires another model step, so long responses can be slow even when the prompt has already been processed.
Speculative decoding tries to generate several tokens per expensive target-model step. A smaller or cheaper draft model proposes a short run of tokens. The full target model checks those proposals in parallel and accepts as many as its own probability distribution allows.
When many proposed tokens are accepted, the target model advances several positions at once. When a proposal is rejected, generation falls back to the target model at that position.
Draft, verify, accept
The basic process is:
- The draft model proposes several future tokens.
- The target model scores those positions in one verification pass.
- An acceptance rule keeps a valid prefix of the draft.
- At the first rejection, a replacement token is sampled from a corrected target distribution.
- The process repeats from the new prefix.
The exact acceptance math matters. Correct speculative sampling is designed to preserve the target model’s output distribution. It is not simply “use the draft model when it looks confident.”
Some systems use a small separate model. Others use extra prediction heads, an early-exit path, or a tree of candidate continuations. The shared idea is cheap proposal followed by authoritative verification.
A simple example
Suppose the current text is:
The deployment failed because
The draft model proposes:
the database connection timed out
The target model verifies all five proposed positions. It accepts:
the database connection
At the next position, the target distribution does not accept timed. The algorithm samples a replacement such as was, producing:
The deployment failed because the database connection was
The draft model then proposes again from that updated prefix.
The target model remains authoritative. The draft only suggests work that may be accepted.
Why verification can be faster
During normal inference, decode is sequential: one target-model step produces one new token.
Speculative decoding uses extra compute from the draft model but may reduce the number of sequential target-model steps. Modern accelerators can score several proposed positions in one pass more efficiently than running the target model separately for each position.
The speedup depends on:
- How cheap the draft is compared with the target.
- How many draft tokens are proposed at once.
- How often the target accepts them.
- Batch size and server load.
- Memory bandwidth and kernel efficiency.
- The cost of moving cache state and synchronizing models.
If acceptance is low, the system pays for draft work that is frequently discarded.
Acceptance rate
Acceptance rate is the share of proposed tokens that the target accepts. A draft model that predicts similarly to the target tends to achieve a higher rate.
Acceptance can fall when:
- The prompt asks for a rare domain or language.
- Sampling temperature is high.
- The target model is much stronger than the draft.
- The draft uses a different tokenizer or incompatible vocabulary.
- The text changes direction frequently, such as creative writing.
A larger draft may improve acceptance but also cost more. The best draft is not necessarily the smallest or most accurate model. It is the model that produces useful accepted tokens for the lowest total serving cost.
Output quality and exactness
There are two different goals:
- Distribution-preserving sampling: the algorithm produces samples from the same target distribution, within the assumptions and numerical behavior of the implementation.
- Bit-for-bit identical output: the exact same random sequence and floating-point execution produce identical tokens.
The first is the usual theoretical goal. The second is harder in real systems because parallel execution, random-number consumption, kernels, and numeric precision can differ.
For greedy decoding, implementations can often verify draft tokens against the target’s highest-probability token. For sampling, the acceptance and correction rules are more involved.
Do not assume every feature called “speculative decoding” preserves output behavior in the same way. Check the serving engine’s algorithm and evaluation.
KV caches and memory
Both draft and target models need state for the current prefix. In transformer serving, this usually includes a KV cache for each model.
The system must:
- Add cache entries for accepted tokens.
- Discard or roll back rejected speculative positions.
- Keep tokenization aligned.
- Avoid copying so much cache data that the optimization loses its benefit.
A separate draft model also consumes weights and memory. This can reduce the number of requests that fit on one accelerator.
Where it helps
Speculative decoding is most attractive when:
- The target model is expensive.
- Responses contain many output tokens.
- The draft model is much cheaper.
- Draft and target predictions agree often.
- Decode latency matters more than prompt prefill.
- The serving stack supports efficient verification and cache management.
It may help less when prompts are long and outputs are short, because prefill dominates. It may also help less at high batch sizes if the server already uses the accelerator efficiently.
Evaluating a deployment
Measure the full service, not just accepted tokens:
- Time to first token.
- Inter-token latency.
- Output tokens per second.
- End-to-end latency.
- Acceptance rate and accepted tokens per target step.
- Requests per second at realistic concurrency.
- Accelerator memory use.
- Cost per completed request.
- Output distribution or task quality.
Test representative prompts, languages, temperatures, and output lengths. A benchmark with easy, predictable text can overstate gains for real workloads.
Common misconceptions
- The draft model does not replace the target model.
- Rejected tokens are not shown to the user.
- A high acceptance rate does not guarantee lower cost if the draft is expensive.
- Longer speculative blocks are not always better.
- The optimization mainly targets decode; it does not remove prompt prefill.
- Correct speculative sampling is more than comparing one confidence score.
The key idea
Speculative decoding trades cheap proposal work for fewer sequential target-model steps. It is useful when the draft is inexpensive, agreement is high, and the serving system can verify and manage cache state efficiently without changing the intended target behavior.