Speculative decoding: how drafts speed up LLM inference

Speculative decoding speeds up LLM inference by having a cheap draft propose several tokens that a large target model checks in a single forward pass. With the standard acceptance rule, the output distribution matches sampling from the target alone. The gain depends on acceptance rate, draft cost, and spare GPU compute, and you can measure all three. Engine support also changes fast, and vLLM v0.33.0 removed two draft methods in November 2026.

#What is speculative decoding

Speculative decoding is a way to generate tokens from a large model faster without changing its outputs. A cheap draft proposes several tokens, the large target model verifies them in one forward pass, and every accepted token saves a full target step. It targets decode, where the GPU mostly waits on memory.

During decode, a target model produces one token per forward pass, and each pass reads its weights and its KV cache from GPU memory. The arithmetic units mostly wait for that data, so spare compute is available to verify extra positions in the same pass.

Leviathan et al. posted the method in November 2022, and Chen et al. posted it independently in February 2023 as speculative sampling.

The first paper reports 2x to 3x on T5-XXL, and the second reports 2x to 2.5x on the 70B Chinchilla model. Both figures come from the authors.

#How one round of speculative decoding works

One round drafts k tokens, runs one target pass over the prefix plus all k drafts, and accepts them left to right until one fails. The first rejected position gets a target-sampled token, the rest are discarded, and if all k pass, the target adds one bonus token.

The target model is the large model whose outputs you want. The draft is any cheap source of guesses. Here is one round in order:

1. The draft generates k tokens one at a time.
2. The target runs one forward pass over the prefix plus all k drafts, which gives its probability for every drafted position.
3. Starting at the first draft token, each token is accepted with probability min(1, p(x) / q(x)), where p is the target and q is the draft.
4. At the first rejection, the target samples a replacement from the adjusted distribution, max(0, p - q) renormalized. Later drafts are discarded.
5. If all k tokens pass, the same forward pass already scored position k + 1, so the target adds that token at no extra cost.

The paper's Algorithm 1 returns between 1 and k + 1 tokens per round. The cost is one target pass plus k draft steps.

#Why speculative decoding preserves outputs

The method preserves the target's output distribution because its acceptance rule is rejection sampling. A draft token x is kept with probability min(1, p(x) / q(x)), and each rejection draws from the leftover probability mass, which reproduces p exactly. Chen et al. describe the same guarantee, within hardware numerics.

A small simulation makes the guarantee and Equation 1 concrete. It uses hand-built next-token tables as the target and draft, samples 400,000 six-token sequences, and compares them with plain sampling from the target. No LLM or GPU ran, so this tests the rule and the formula and says nothing about any model's real acceptance rate.

"""Speculative sampling on a toy Markov source. Python stdlib only.
 
No model runs here: TARGET and DRAFT are hand-built next-token tables.
It tests the acceptance rule and Equation 1, not any real model's alpha.
"""
import random
from collections import Counter
 
TOKENS = "abcd"
 
 
def table(seed):
    rng = random.Random(seed)
    tbl = {}
    for ctx in TOKENS:
        w = [rng.random() + 0.05 for _ in TOKENS]
        tbl[ctx] = {t: v / sum(w) for t, v in zip(TOKENS, w)}
    return tbl
 
 
TARGET = table(7)
NOISE = table(11)
DRAFT = {c: {t: 0.3 * TARGET[c][t] + 0.7 * NOISE[c][t] for t in TOKENS} for c in TOKENS}
 
 
def draw(dist, rng):
    x, total = rng.random(), 0.0
    for token, p in dist.items():
        total += p
        if x < total:
            return token
    return token
 
 
def residual(p, q):
    left = {t: max(0.0, p[t] - q[t]) for t in p}
    norm = sum(left.values())
    return {t: v / norm for t, v in left.items()}
 
 
def spec_round(last, k, rng, stats):
    """One target pass over k drafted tokens."""
    drafts, ctx = [], last
    for _ in range(k):
        token = draw(DRAFT[ctx], rng)
        drafts.append((ctx, token))
        ctx = token
    stats["passes"] += 1
    out = []
    for ctx, token in drafts:
        p, q = TARGET[ctx], DRAFT[ctx]
        accept = min(1.0, p[token] / q[token])
        stats["accept_sum"] += accept
        stats["checked"] += 1
        if rng.random() < accept:
            out.append(token)
        else:
            out.append(draw(residual(p, q), rng))  # the corrected token
            return out
    out.append(draw(TARGET[drafts[-1][1]], rng))  # the bonus token
    return out
 
 
def speculative(n, k, seed, length=6):
    rng, stats, seqs, made = random.Random(seed), Counter(), Counter(), 0
    for _ in range(n):
        seq, last = "", "a"
        while len(seq) < length:
            new = spec_round(last, k, rng, stats)
            seq += "".join(new)
            made += len(new)
            last = seq[-1]
        seqs[seq[:length]] += 1
    alpha = stats["accept_sum"] / stats["checked"]
    per_pass = made / stats["passes"]
    return seqs, alpha, per_pass
 
 
def plain(n, seed, length=6):
    rng, seqs = random.Random(seed), Counter()
    for _ in range(n):
        seq, last = "", "a"
        while len(seq) < length:
            last = draw(TARGET[last], rng)
            seq += last
        seqs[seq] += 1
    return seqs
 
 
def tv(a, b, n):
    return 0.5 * sum(abs(a[s] - b[s]) / n for s in set(a) | set(b))
 
 
N = 400_000
ref = plain(N, seed=1)
noise_floor = tv(ref, plain(N, seed=2), N)
print(f"total variation, plain vs plain:       {noise_floor:.4f}")
for k in (1, 2, 4, 8):
    seqs, alpha, per_pass = speculative(N, k, seed=3 + k)
    eq1 = (1 - alpha ** (k + 1)) / (1 - alpha)
    print(f"k={k}: TV vs plain {tv(ref, seqs, N):.4f}, "
          f"measured alpha {alpha:.3f}, tokens per pass {per_pass:.3f}, Eq. 1 {eq1:.3f}")

total variation, plain vs plain:       0.0421
k=1: TV vs plain 0.0418, measured alpha 0.858, tokens per pass 1.858, Eq. 1 1.858
k=2: TV vs plain 0.0430, measured alpha 0.856, tokens per pass 2.589, Eq. 1 2.588
k=4: TV vs plain 0.0426, measured alpha 0.857, tokens per pass 3.755, Eq. 1 3.760
k=8: TV vs plain 0.0428, measured alpha 0.857, tokens per pass 5.248, Eq. 1 5.257

Total variation distance stays at the 0.042 noise floor of two plain sampling runs, and tokens per pass match Equation 1 to within 0.01.

#How to estimate speculative decoding speedup

Tokens per target pass come from Equation 1 in Leviathan et al., (1 - alpha^(k + 1)) / (1 - alpha). Divide by k * c + 1, where c is the draft's cost relative to one target step, to get the walltime speedup in Theorem 3.8. Both assume acceptance is independent across positions.

def tokens_per_pass(alpha: float, k: int) -> float:
    return (1 - alpha ** (k + 1)) / (1 - alpha)
 
def speedup(alpha: float, k: int, c: float) -> float:
    return tokens_per_pass(alpha, k) / (k * c + 1)

For k = 4, recomputed with code on November 8, 2026:

Treat these as ceilings. The paper calls the independence assumption an approximation, and its walltime analysis assumes the target can verify k + 1 positions in parallel without slowing down. A slow draft eats most of the gain: a draft at one third of a target step, with 0.5 acceptance, runs slower than no speculation at all.

The best draft length depends on acceptance and cost, and the paper finds it numerically.

At alpha 0.8, a drafter with c = 0.05 is fastest at k = 8 (3.09x), and one with c = 0.33 is fastest at k = 3 (1.48x). Going from k = 4 to k = 8 at alpha 0.7 adds only 0.43 expected tokens per pass while doubling draft work.

#Where draft tokens come from

Draft tokens come from a separate small model, a small network on the target's hidden features, multi-token heads trained with the target, a block-diffusion drafter, or lookup in text already seen. They trade draft quality against draft cost, the c in the speedup formula.

#A separate small model

The original setup pairs a small model with a much larger sibling. vLLM's draft model docs use Qwen3-0.6B as a draft for Qwen3-8B. The draft needs its own weights and its own KV cache, which raises c and memory use.

#Hidden features (EAGLE)

EAGLE drafts from the target's hidden features instead of tokens, using a small layer trained on the target. Its authors report about 0.8 acceptance against about 0.6 for Medusa, which adds extra prediction heads to the target. Those are author-reported figures on the paper's benchmarks.

#Multi-token prediction (MTP)

MTP heads are trained with the target itself, so the checkpoint already carries a draft. DeepSeek-V3 trained them and notes they can be reused for speculative decoding. vLLM says MTP only works for model families it supports.

#Block diffusion

DFlash proposes a whole block of tokens in one pass with a small diffusion model conditioned on the target's features. Its authors report over 6x lossless acceleration and up to 2.5x the speedup of EAGLE-3, on their own benchmarks. vLLM's v0.33.0 notes describe DSpark as a similar block-diffusion drafter.

#N-gram and suffix lookup

Lookup methods copy from text already seen, so they need no model. They work best when output repeats input, such as code edits and summarization.

vLLM measured up to 2.8x with prompt lookup on CNN/DailyMail at QPS 1, in its own 2024 benchmark. Best LLM for coding covers the model choice for that workload.

#What vLLM and SGLang support today

As of November 8, 2026, vLLM's speculative_config accepts ngram, ngram_gpu, draft_model, suffix, eagle, eagle3, mtp, dflash, dspark, and extract_hidden_states. SGLang documents EAGLE, EAGLE3, STANDALONE, NGRAM, DFLASH, and UNO. Medusa is in neither list, because vLLM removed it in v0.33.0 and SGLang's docs do not mention it.

The method names come from vLLM's config source, its docs, and SGLang's docs. None of these flags ran for this post.

vLLM merged the Medusa removal on October 15, 2026. The pull request says Medusa was not supported by the newer model runner, the MLP speculator was already disabled, and EAGLE, MTP, DFlash, and DSpark cover the use case. The v0.33.0 release notes list it under breaking changes.

Those notes sit on the tag v0.33.0rc3, published November 2, 2026, while the bare v0.33.0 tag page has no notes. A config that still sets method="medusa" now fails validation.

The newest vLLM tag on November 8, 2026 is still v0.33.0, and main does not bring either method back. The vLLM, SGLang, or llama.cpp comparison covers the engines more broadly.

#When speculative decoding hurts

Speculative decoding hurts when the GPU has no compute to spare, when acceptance is low, or when the draft costs too much. vLLM's own 2024 benchmarks showed 1.4x and 1.8x slowdowns on Llama 3 70B at high QPS, and its docs now aim the feature at medium-to-low QPS, memory-bound workloads.

#Load

vLLM's 2024 post reported up to 1.5x on ShareGPT at QPS 1 and the slowdowns above at high QPS, on four H100s, with no independent measurement. Verification multiplies the effective batch: vLLM's dynamic speculative decoding docs say that once batch size times K passes a critical point, decode speed drops. Their fix is a per-concurrency K, with [[1, 64, 3], [65, 128, 1], [129, 512, 0]] as the example.

#Low acceptance

High temperature and open-ended prose are harder to predict. At an alpha of 0.3 and k = 4, Equation 1 gives 1.43 tokens per pass, and a draft at c = 0.10 brings the speedup to 1.02x. Most draft work is thrown away at that rate.

#Memory for the draft

A separate draft adds its own weights and its own KV cache, which vLLM's memory manager now tracks alongside the target's. MTP and EAGLE heads avoid a whole second model, and lookup methods add no weights.

#Lossy verification

Some speedups give up the distribution guarantee on purpose. vLLM's FLy verification defers some rejections and is documented as intentionally lossy, with 1.04x to 1.31x decode throughput reported on MI355X and B300 in vLLM's own release notes. Use it only if you test output quality on your task.

#How to measure speculative decoding on your traffic

Measure acceptance, per-token latency, and the batch sizes you actually run. vLLM exposes vllm:spec_decode_num_draft_tokens and vllm:spec_decode_num_accepted_tokens at /metrics, and their ratio is your acceptance rate. Compare time per output token with speculation on and off at your real concurrency.

The metric names come from vLLM's source. The per-request flag --per-request-spec-decode-metrics summary reports a mean_acceptance_length under metrics.speculative_decoding, which is experimental and may change between releases.

Run the speedup function with your measured alpha and c before you pay for a new draft model. Watch throughput as well as time to first token, because a high-QPS endpoint can lose throughput while a low-load test still looks fine.

#How to decide on speculative decoding

Turn it on for latency-sensitive, low-concurrency endpoints first, such as interactive chat and code completion. Leave high-throughput batch jobs alone until a load test shows a gain. Use this order when you pick a method, and retest after each engine upgrade.

1. If your model ships MTP heads or an EAGLE checkpoint, start there, because the draft is trained on the target and adds little memory.
2. If your prompts repeat in the output, enable ngram or suffix, because they cost almost no compute.
3. If neither applies, try a small draft from the same model family and measure alpha, c, and memory.
4. If you expect high concurrency, set a per-concurrency K of zero at the top of the range.

The KV cache budget and batching strategy sit underneath this choice, and a sparse mixture-of-experts model changes that math again. The inference topic page collects the related guides.

#Questions