Speculative Decoding for LLM Inference: How Draft-and-Verify Accelerates Token Generation

By kongastral

Published August 23, 2026 · 18 min read

Standard autoregressive text generation produces one token per forward pass through the network, and a language model must repeat that pass once for every token it emits. Speculative decoding replaces this strictly sequential loop with a draft-and-verify scheme: a small, fast model proposes a short run of tokens at once, and the large model checks all of them in a single pass. The two approaches produce the same text, but they spend the expensive model’s time very differently. Understanding why the second approach can be several times faster, and why it is not free, requires looking at what actually limits the speed of a forward pass.

The technique was introduced independently in two 2023 papers and has since become a standard component of production inference stacks. This guide explains the mechanism, the mathematics that make it lossless, the speedups that have been measured, the main variants that avoid a separate draft model, and the regime in which the method pays off.

Why autoregressive decoding is slow

A transformer decoder generates text one token at a time. Each new token is conditioned on all previous tokens, so the model runs a full forward pass, produces a probability distribution over the vocabulary, samples a token, appends it, and repeats. For a response of several hundred tokens, the large model is invoked several hundred times in strict sequence. This dependency is the reason latency scales with output length.

The important detail is that a single-token forward pass does not use the accelerator’s arithmetic units efficiently. Generating one token requires reading the model’s weights from memory but performs relatively little computation per byte read, so the pass is memory-bandwidth-bound: its wall-clock time is dominated by moving weights, not by multiplying numbers. A modern accelerator therefore has spare arithmetic capacity during each decode step. Chen and colleagues at DeepMind framed the consequence precisely: scoring a short continuation of several tokens in parallel has latency comparable to sampling a single token, because the extra tokens ride along in the same memory-bound pass (Chen et al., 2023). Speculative decoding is the technique that turns that spare capacity into useful output.

The draft-and-verify loop

Speculative decoding pairs two models. The target model is the large, accurate network whose output is desired. The draft model is a much smaller network that approximates the target and can run many times faster. A single iteration proceeds in three steps.

First, the draft model generates a short candidate continuation autoregressively, proposing a fixed number of tokens, conventionally written as the draft length. Because the draft is small, these steps are cheap. Second, the target model processes the original context together with all proposed tokens in one parallel forward pass, yielding its own probability for each position. Third, a rejection rule compares the two models position by position and accepts the longest prefix of proposed tokens that is consistent with the target distribution. When a proposed token is rejected, it is replaced by a token resampled from an adjusted distribution, and the iteration ends. When every proposed token is accepted, the target contributes one additional “bonus” token for free from the same pass.

The number of tokens produced per iteration is therefore variable: it ranges from one, when the very first proposal is rejected, up to the draft length plus one, when all proposals are accepted. The average over many iterations determines the speedup. Each iteration costs one target pass plus the cheap draft steps, and it advances the sequence by more than one token whenever the draft agrees with the target, which is where the acceleration comes from.

The correctness guarantee

The property that distinguishes speculative decoding from lossy speed tricks is that its output is statistically identical to sampling from the target model alone. This is achieved by a modified rejection-sampling rule rather than by trusting the draft. Let the target model assign probability p(x) to a token and the draft model assign it probability q(x). A proposed token drawn from the draft is accepted with probability min(1, p(x)/q(x)) (Leviathan et al., 2023). In words, if the target likes the token at least as much as the draft did, it is always kept; if the target likes it less, it is kept with a probability equal to the ratio of the two.

When a token is rejected, the algorithm does not simply stop with the draft’s choice. It resamples from an adjusted distribution defined as the normalized positive difference between the two, p′(x) = norm(max(0, p(x) − q(x))) (Leviathan et al., 2023). This correction exactly compensates for the cases the draft over-sampled, so that the overall probability of emitting any given token equals the target’s probability. The result is that speculative decoding preserves the target distribution within hardware numerics; it is not an approximation that trades quality for speed, and it needs no fine-tuning or architectural change to the target model.

Because the correctness rule depends only on the two probability values at each position, the guarantee holds regardless of how good or bad the draft model is. A poor draft does not corrupt the output; it merely gets rejected more often and delivers less speedup. This separation between correctness and performance is what makes the method safe to deploy: the worst case is slower generation, never wrong generation. Readers interested in how models behave at the level of output probabilities may find the discussion in classifier calibration and reliability diagrams a useful companion, since both topics turn on treating a model’s probabilities as first-class objects.

How much speedup to expect

The performance of speculative decoding is governed by the acceptance rate, written α, which is the expected probability that a proposed token is accepted. A higher acceptance rate means the draft and target agree more often, so more tokens survive each verification pass. The acceptance rate can be written as α = E(min(p, q)), an expectation that increases as the draft distribution moves closer to the target distribution (Leviathan et al., 2023).

Given an acceptance rate and a draft length γ, the expected number of tokens produced per iteration follows a capped geometric expression:

E[tokens per iteration] = (1 - alpha^(gamma + 1)) / (1 - alpha)

This formula counts the accepted prefix plus the one bonus token the target contributes. As an illustration, with an acceptance rate of 0.7 and a draft length of 4, the expected yield is (1 − 0.7⁵)/(1 − 0.7) ≈ 2.8 tokens per iteration, meaning the sequence advances by nearly three tokens for each expensive target pass. Raising the acceptance rate to 0.9 lifts the expectation to roughly 4.1 tokens per iteration. These are expected token counts, not wall-clock speedups: the realized speedup also depends on the ratio between the draft model’s cost and the target model’s cost, because each iteration must pay for the draft steps as well.

Measured end-to-end results are consistent with this picture. On the T5-XXL model, the original method reaches a 2×–3× wall-clock speedup over the standard implementation while producing identical samples (Leviathan et al., 2023). In a separate study, a 2×–2.5× speedup was measured when sampling from Chinchilla, a 70-billion-parameter model, in a distributed setup, again without degrading sample quality (Chen et al., 2023). A useful rule of thumb, stated cautiously, is that speculative decoding delivers roughly a twofold to threefold latency reduction in the latency-bound settings these papers examined, with the exact figure depending on the draft-target pair and the workload.

The variant landscape

The classic formulation needs a separate draft model whose tokenizer and vocabulary match the target. Finding or training a good draft is the main practical friction, and a second model consumes additional memory. A family of variants addresses this by generating draft tokens without a distinct second network, trading setup complexity for either extra trained components or a narrower range of gains.

Medusa, an approach that attaches extra decoding heads to the target model and verifies their proposals with tree-structured attention, records a 2.2×–3.6× speedup across a range of models and needs no separate draft network (Cai et al., 2024). EAGLE instead performs the draft step at the feature level, autoregressing over the target’s second-to-top-layer representations; on the MT-bench evaluation it runs roughly 3× faster than vanilla decoding while preserving the output distribution (Li et al., 2024). Its successor, EAGLE-2, extends this with dynamically constructed draft trees and records speedup ratios of 3.05×–4.26×, about 20%–40% above the first version, and remains lossless (Li et al., 2024). Lookahead decoding, an exact method motivated by Jacobi iteration that requires no draft model or datastore, reaches up to 1.8× on MT-bench and up to roughly 4× with strong scaling across multiple GPUs on code-completion workloads (Fu et al., 2024). A further line of work draws candidate continuations directly from the prompt or prior output through an n-gram lookup table rather than any neural drafter; it adds no model weights and is most effective on input-grounded, repetitive tasks such as summarization or code editing, where the output often echoes the input.

The variant landscape reflects one underlying trade. Methods on the left of the map require nothing extra but accept fewer tokens per pass; methods on the right invest in trained heads or feature modules to raise the acceptance rate and, with it, the ceiling on speedup. Feature-level and attention-based drafting build on the same transformer machinery covered in the guide to graph attention networks, and the large targets these methods accelerate are the models produced by pretraining regimes such as those described in the overview of self-supervised learning.

When it helps, and when it does not

The single most important practical caveat concerns batch size. Speculative decoding converts spare arithmetic capacity into extra tokens, and that spare capacity exists only when the target’s forward pass is memory-bandwidth-bound. This condition holds in the low-batch, latency-bound regime, where a server processes one request or a few concurrent requests and the accelerator’s compute units are underused. In that regime, verifying several proposed tokens is nearly free, and the technique delivers its full benefit.

At high batch sizes, the situation inverts. When many requests are processed together, the target’s forward pass already keeps the arithmetic units busy, so it becomes compute-bound rather than memory-bound. The extra verification work now competes with real work instead of filling idle capacity, and rejected tokens represent wasted computation that reduces overall throughput. Consequently, speculative decoding is best understood as a latency-optimization technique for interactive, low-concurrency serving rather than a throughput free lunch for saturated batch workloads.

Two further trade-offs shape a deployment. The acceptance rate depends on how well the draft is aligned with the target, so a mismatched draft wastes target compute on rejected proposals; selecting or training a well-aligned draft is the main tuning lever. And the draft length sets a ceiling on tokens per iteration but also raises the wasted work when proposals are rejected, so there is an optimum draft length that depends on the acceptance rate and on the draft-to-target cost ratio.

Support in production systems

Speculative decoding is available in the major inference stacks, though the exact configuration options and flag names change between releases and should be checked against current documentation rather than memorized. The mechanism, however, is stable across them.

The vLLM serving engine supports several drafters, including zero-overhead n-gram lookup for repetitive workloads and neural methods such as Medusa and the EAGLE family for higher acceptance. NVIDIA’s TensorRT-LLM similarly exposes the draft-model approach alongside EAGLE, Lookahead, Medusa, and related drafters executed within its compiled engine. In the Hugging Face transformers library, the feature is called assisted generation: passing an assistant model to the generation call causes the assistant to draft tokens that the main model verifies in a single pass, and a universal variant relaxes the requirement that the two models share a tokenizer. The C and C++ project llama.cpp enables the same idea by supplying a draft model to its command-line and server tools, provided the draft shares a compatible vocabulary with the main model. Deploying any of these behind a service boundary raises the usual operational questions covered in the guides to containerizing applications for production and to running stateful workloads on Kubernetes pods.

The common thread across these systems is that speculative decoding is applied at inference time as a drop-in accelerator: it changes how tokens are produced, not what the target model is, and its lossless guarantee means an existing model can be served faster without retraining or quality regression. The main engineering decisions are which drafting strategy to use, how to size the draft, and whether the serving regime is latency-bound enough for the technique to pay off.

Conclusion

Speculative decoding accelerates language model inference by exploiting a structural fact: a single-token decode step leaves the accelerator’s arithmetic units idle, and that idle capacity can verify several draft tokens at almost no extra cost. A modified rejection-sampling rule, accepting each proposed token with probability min(1, p/q) and resampling from an adjusted distribution on rejection, makes the acceleration lossless within hardware numerics rather than an approximation. The expected yield per iteration grows with the acceptance rate through the expression (1 − α^γ+1)/(1 − α), and measured latency reductions in the range of two to three times have been reported for the settings the founding papers studied.

The variant landscape, from n-gram lookup through Lookahead, Medusa, and the EAGLE family, trades setup cost against acceptance rate, letting a practitioner pick a point that fits the deployment. The one caveat that determines whether the technique is worthwhile is the serving regime: it is a strong latency optimization for low-batch, interactive inference and a weak one for saturated high-throughput batches. Read against those constraints, speculative decoding is a well-understood and broadly supported way to serve an existing model faster without changing what it produces.

References

• Leviathan, Y., Kalman, M., Matias, Y. “Fast Inference from Transformers via Speculative Decoding.” ICML 2023. arxiv.org/abs/2211.17192
• Chen, C., et al. (DeepMind). “Accelerating Large Language Model Decoding with Speculative Sampling.” 2023. arxiv.org/abs/2302.01318
• Cai, T., et al. “Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads.” 2024. arxiv.org/abs/2401.10774
• Li, Y., et al. “EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees.” 2024. arxiv.org/abs/2406.16858 (EAGLE: arxiv.org/abs/2401.15077)
• Fu, Y., et al. “Break the Sequential Dependency of LLM Inference Using Lookahead Decoding.” ICML 2024. arxiv.org/abs/2402.02057

Frequently Asked Questions

Does speculative decoding change the text a model produces?

No. The modified rejection-sampling rule preserves the target model’s output distribution within hardware numerics, so the generated text is statistically identical to standard decoding from the target. The draft model only affects speed, never correctness; a poor draft slows generation but cannot change what is produced.

How large a speedup is realistic?

The founding papers reported roughly two-to-threefold latency reductions in latency-bound settings, such as 2×–3× on T5-XXL and 2×–2.5× on a 70-billion-parameter model. The realized figure depends on the acceptance rate between draft and target, the draft length, and the cost ratio of the two models, so results vary by workload.

What is the acceptance rate and why does it matter?

The acceptance rate is the expected probability that a proposed draft token is kept by the target. It rises as the draft distribution approaches the target distribution. Because the expected tokens per iteration equal (1 − α^(γ+1))/(1 − α), a higher acceptance rate directly increases how many tokens each expensive target pass yields.

Do all variants require a separate draft model?

No. The classic method uses a separate small model, but Medusa adds decoding heads to the target, EAGLE drafts at the feature level, Lookahead decoding uses Jacobi-style n-gram trajectories with no draft model, and prompt or n-gram lookup copies candidates from the input. These variants avoid a second model at the cost of trained components or a narrower range of gains.

Why does the benefit shrink at high batch sizes?

The technique fills spare arithmetic capacity that exists only when the target’s forward pass is memory-bandwidth-bound, which is the case at low batch sizes. At high batch sizes the pass becomes compute-bound, the spare capacity disappears, and the extra verification work competes with real work. Speculative decoding is therefore most useful for interactive, low-concurrency serving.