---
title: "Speculative decoding: how drafts speed up LLM inference"
type: "deep dive"
url: "https://aifordevs.org/articles/speculative-decoding"
markdown: "https://aifordevs.org/articles/speculative-decoding.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/articles/speculative-decoding"
description: "Speculative decoding has a cheap draft propose tokens the target model verifies in one pass. See the math, the costs, and what vLLM and SGLang support."
author: "AI for Devs"
published: "2026-11-06"
updated: "2026-11-08"
level: "Advanced"
topics: ["inference"]
---

# Speculative decoding: how drafts speed up LLM inference

Speculative decoding has a cheap draft propose tokens the target model verifies in one pass. See the math, the costs, and what vLLM and SGLang support.

> **TL;DR**
>
> - Speculative decoding has a cheap draft propose tokens that the target model verifies in one pass, and the standard acceptance rule keeps the output distribution identical to the target alone.
> - Expected tokens per target pass equal `(1 - alpha^(k + 1)) / (1 - alpha)`, Equation 1 in Leviathan et al. At alpha 0.8 and k 4 that is 3.36, and a draft costing 10 percent of a target step cuts the walltime gain to 2.40 times.
> - vLLM removed Medusa and the MLP speculator in v0.33.0, published November 2, 2026, and no newer release has reversed that as of November 8, 2026.
> - vLLM and SGLang both document EAGLE, multi-token prediction, block-diffusion, and n-gram drafts today, under different flag names.
> - The simulation in this article ran on Python 3.15.1 with hand-built tables. No LLM, GPU, vLLM, or SGLang process ran for this post.

Speculative decoding speeds up LLM [inference](https://aifordevs.org/glossary#inference) by having a cheap draft propose several tokens that a large target model checks in a single forward pass. With the standard acceptance rule, the output distribution matches sampling from the target alone. The gain depends on acceptance rate, draft cost, and spare GPU compute, and you can measure all three. Engine support also changes fast, and vLLM v0.33.0 removed two draft methods in November 2026.

## What is speculative decoding

Speculative decoding is a way to generate tokens from a large model faster without changing its outputs. A cheap draft proposes several tokens, the large target model verifies them in one forward pass, and every accepted token saves a full target step. It targets [decode](https://aifordevs.org/glossary#prefill-and-decode), where the GPU mostly waits on memory.

During decode, a target model produces one [token](https://aifordevs.org/glossary#token) per forward pass, and each pass reads its weights and its [KV cache](https://aifordevs.org/articles/kv-cache-explained) from GPU memory. The arithmetic units mostly wait for that data, so spare compute is available to verify extra positions in the same pass.

[Leviathan et al.](https://arxiv.org/abs/2211.17192) posted the method in November 2022, and [Chen et al.](https://arxiv.org/abs/2302.01318) posted it independently in February 2023 as speculative sampling.

The first paper reports 2x to 3x on T5-XXL, and the second reports 2x to 2.5x on the 70B Chinchilla model. Both figures come from the authors.

## How one round of speculative decoding works

One round drafts k tokens, runs one target pass over the prefix plus all k drafts, and accepts them left to right until one fails. The first rejected position gets a target-sampled token, the rest are discarded, and if all k pass, the target adds one bonus token.

**Figure 1.** The draft proposes four tokens. One target pass scores all four positions, accepts the first two, rejects the third, and samples its own token in that position. The fourth draft token is never checked.

The target model is the large model whose outputs you want. The draft is any cheap source of guesses. Here is one round in order:

1. The draft generates k tokens one at a time.
2. The target runs one forward pass over the prefix plus all k drafts, which gives its probability for every drafted position.
3. Starting at the first draft token, each token is accepted with probability `min(1, p(x) / q(x))`, where `p` is the target and `q` is the draft.
4. At the first rejection, the target samples a replacement from the adjusted distribution, `max(0, p - q)` renormalized. Later drafts are discarded.
5. If all k tokens pass, the same forward pass already scored position k + 1, so the target adds that token at no extra cost.

The paper's [Algorithm 1](https://arxiv.org/abs/2211.17192) returns between 1 and k + 1 tokens per round. The cost is one target pass plus k draft steps.

## Why speculative decoding preserves outputs

The method preserves the target's output distribution because its acceptance rule is rejection sampling. A draft token `x` is kept with probability `min(1, p(x) / q(x))`, and each rejection draws from the leftover probability mass, which reproduces `p` exactly. Chen et al. describe the same guarantee, within hardware numerics.

A small simulation makes the guarantee and Equation 1 concrete. It uses hand-built next-token tables as the target and draft, samples 400,000 six-token sequences, and compares them with plain sampling from the target. No LLM or GPU ran, so this tests the rule and the formula and says nothing about any model's real acceptance rate.

```python title="spec_sim.py"
"""Speculative sampling on a toy Markov source. Python stdlib only.

No model runs here: TARGET and DRAFT are hand-built next-token tables.
It tests the acceptance rule and Equation 1, not any real model's alpha.
"""
import random
from collections import Counter

TOKENS = "abcd"


def table(seed):
    rng = random.Random(seed)
    tbl = {}
    for ctx in TOKENS:
        w = [rng.random() + 0.05 for _ in TOKENS]
        tbl[ctx] = {t: v / sum(w) for t, v in zip(TOKENS, w)}
    return tbl


TARGET = table(7)
NOISE = table(11)
DRAFT = {c: {t: 0.3 * TARGET[c][t] + 0.7 * NOISE[c][t] for t in TOKENS} for c in TOKENS}


def draw(dist, rng):
    x, total = rng.random(), 0.0
    for token, p in dist.items():
        total += p
        if x < total:
            return token
    return token


def residual(p, q):
    left = {t: max(0.0, p[t] - q[t]) for t in p}
    norm = sum(left.values())
    return {t: v / norm for t, v in left.items()}


def spec_round(last, k, rng, stats):
    """One target pass over k drafted tokens."""
    drafts, ctx = [], last
    for _ in range(k):
        token = draw(DRAFT[ctx], rng)
        drafts.append((ctx, token))
        ctx = token
    stats["passes"] += 1
    out = []
    for ctx, token in drafts:
        p, q = TARGET[ctx], DRAFT[ctx]
        accept = min(1.0, p[token] / q[token])
        stats["accept_sum"] += accept
        stats["checked"] += 1
        if rng.random() < accept:
            out.append(token)
        else:
            out.append(draw(residual(p, q), rng))  # the corrected token
            return out
    out.append(draw(TARGET[drafts[-1][1]], rng))  # the bonus token
    return out


def speculative(n, k, seed, length=6):
    rng, stats, seqs, made = random.Random(seed), Counter(), Counter(), 0
    for _ in range(n):
        seq, last = "", "a"
        while len(seq) < length:
            new = spec_round(last, k, rng, stats)
            seq += "".join(new)
            made += len(new)
            last = seq[-1]
        seqs[seq[:length]] += 1
    alpha = stats["accept_sum"] / stats["checked"]
    per_pass = made / stats["passes"]
    return seqs, alpha, per_pass


def plain(n, seed, length=6):
    rng, seqs = random.Random(seed), Counter()
    for _ in range(n):
        seq, last = "", "a"
        while len(seq) < length:
            last = draw(TARGET[last], rng)
            seq += last
        seqs[seq] += 1
    return seqs


def tv(a, b, n):
    return 0.5 * sum(abs(a[s] - b[s]) / n for s in set(a) | set(b))


N = 400_000
ref = plain(N, seed=1)
noise_floor = tv(ref, plain(N, seed=2), N)
print(f"total variation, plain vs plain:       {noise_floor:.4f}")
for k in (1, 2, 4, 8):
    seqs, alpha, per_pass = speculative(N, k, seed=3 + k)
    eq1 = (1 - alpha ** (k + 1)) / (1 - alpha)
    print(f"k={k}: TV vs plain {tv(ref, seqs, N):.4f}, "
          f"measured alpha {alpha:.3f}, tokens per pass {per_pass:.3f}, Eq. 1 {eq1:.3f}")
```

```text
total variation, plain vs plain:       0.0421
k=1: TV vs plain 0.0418, measured alpha 0.858, tokens per pass 1.858, Eq. 1 1.858
k=2: TV vs plain 0.0430, measured alpha 0.856, tokens per pass 2.589, Eq. 1 2.588
k=4: TV vs plain 0.0426, measured alpha 0.857, tokens per pass 3.755, Eq. 1 3.760
k=8: TV vs plain 0.0428, measured alpha 0.857, tokens per pass 5.248, Eq. 1 5.257
```

Total variation distance stays at the 0.042 noise floor of two plain sampling runs, and tokens per pass match Equation 1 to within 0.01.

## How to estimate speculative decoding speedup

Tokens per target pass come from Equation 1 in Leviathan et al., `(1 - alpha^(k + 1)) / (1 - alpha)`. Divide by `k * c + 1`, where `c` is the draft's cost relative to one target step, to get the walltime speedup in Theorem 3.8. Both assume acceptance is independent across positions.

```python
def tokens_per_pass(alpha: float, k: int) -> float:
    return (1 - alpha ** (k + 1)) / (1 - alpha)

def speedup(alpha: float, k: int, c: float) -> float:
    return tokens_per_pass(alpha, k) / (k * c + 1)
```

For k = 4, recomputed with code on November 8, 2026:

| Acceptance rate | Tokens per pass | Speedup at c = 0.02 | c = 0.05 | c = 0.10 | c = 0.33 |
| --- | --- | --- | --- | --- | --- |
| 0.5 | 1.94 | 1.79x | 1.61x | 1.38x | 0.84x |
| 0.6 | 2.31 | 2.13x | 1.92x | 1.65x | 0.99x |
| 0.7 | 2.77 | 2.57x | 2.31x | 1.98x | 1.20x |
| 0.8 | 3.36 | 3.11x | 2.80x | 2.40x | 1.45x |
| 0.9 | 4.10 | 3.79x | 3.41x | 2.93x | 1.77x |

Treat these as ceilings. The paper calls the independence assumption an approximation, and its walltime analysis assumes the target can verify k + 1 positions in parallel without slowing down. A slow draft eats most of the gain: a draft at one third of a target step, with 0.5 acceptance, runs slower than no speculation at all.

The best draft length depends on acceptance and cost, and the paper finds it numerically.

**Figure 2.** At 0.7 acceptance, tokens per pass keep climbing with draft length, but once you pay for the draft the speedup peaks and then falls. The best k shrinks as the draft gets more expensive.

At alpha 0.8, a drafter with c = 0.05 is fastest at k = 8 (3.09x), and one with c = 0.33 is fastest at k = 3 (1.48x). Going from k = 4 to k = 8 at alpha 0.7 adds only 0.43 expected tokens per pass while doubling draft work.

## Where draft tokens come from

Draft tokens come from a separate small model, a small network on the target's hidden features, multi-token heads trained with the target, a block-diffusion drafter, or lookup in text already seen. They trade draft quality against draft cost, the `c` in the speedup formula.

### A separate small model

The original setup pairs a small model with a much larger sibling. vLLM's [draft model docs](https://docs.vllm.ai/en/latest/features/speculative_decoding/) use Qwen3-0.6B as a draft for Qwen3-8B. The draft needs its own weights and its own KV cache, which raises `c` and memory use.

### Hidden features (EAGLE)

[EAGLE](https://arxiv.org/abs/2401.15077) drafts from the target's hidden features instead of tokens, using a small layer trained on the target. Its authors report about 0.8 acceptance against about 0.6 for [Medusa](https://arxiv.org/abs/2401.10774), which adds extra prediction heads to the target. Those are author-reported figures on the paper's benchmarks.

### Multi-token prediction (MTP)

MTP heads are trained with the target itself, so the checkpoint already carries a draft. [DeepSeek-V3](https://arxiv.org/abs/2412.19437) trained them and notes they can be reused for speculative decoding. vLLM says [MTP only works for model families it supports](https://docs.vllm.ai/en/latest/features/speculative_decoding/mtp/).

### Block diffusion

[DFlash](https://arxiv.org/abs/2602.06036) proposes a whole block of tokens in one pass with a small diffusion model conditioned on the target's features. Its authors report over 6x lossless acceleration and up to 2.5x the speedup of EAGLE-3, on their own benchmarks. vLLM's [v0.33.0 notes](https://github.com/vllm-project/vllm/releases/tag/v0.33.0rc3) describe DSpark as a similar block-diffusion drafter.

### N-gram and suffix lookup

Lookup methods copy from text already seen, so they need no model. They work best when output repeats input, such as code edits and summarization.

vLLM measured [up to 2.8x with prompt lookup on CNN/DailyMail](https://vllm.ai/blog/2024-10-17-spec-decode) at QPS 1, in its own 2024 benchmark. [Best LLM for coding](https://aifordevs.org/articles/best-llm-for-coding) covers the model choice for that workload.

## What vLLM and SGLang support today

As of November 8, 2026, vLLM's `speculative_config` accepts `ngram`, `ngram_gpu`, `draft_model`, `suffix`, `eagle`, `eagle3`, `mtp`, `dflash`, `dspark`, and `extract_hidden_states`. SGLang documents EAGLE, EAGLE3, STANDALONE, NGRAM, DFLASH, and UNO. Medusa is in neither list, because vLLM removed it in v0.33.0 and SGLang's docs do not mention it.

| Draft source | vLLM `method` | SGLang `--speculative-algorithm` |
| --- | --- | --- |
| Separate small model | `draft_model` | `STANDALONE` |
| EAGLE family | `eagle`, `eagle3` | `EAGLE` (EAGLE-2), `EAGLE3`, `NEXTN` as an alias of `EAGLE` |
| Multi-token heads | `mtp` | `EAGLE` with an MTP checkpoint |
| Block diffusion | `dflash`, `dspark` | `DFLASH` |
| Prompt lookup | `ngram`, `ngram_gpu` | `NGRAM` |
| Earlier responses | `suffix` | Not documented |
| LoRA on the target | Not documented | `UNO` |
| Medusa, MLP speculator | Removed in v0.33.0 | Not documented |

The method names come from [vLLM's config source](https://github.com/vllm-project/vllm/blob/main/vllm/config/speculative.py), [its docs](https://docs.vllm.ai/en/latest/features/speculative_decoding/), and [SGLang's docs](https://docs.sglang.io/docs/advanced_features/speculative_decoding). None of these flags ran for this post.

vLLM merged the [Medusa removal](https://github.com/vllm-project/vllm/pull/61701) on October 15, 2026. The pull request says Medusa was not supported by the newer model runner, the MLP speculator was already disabled, and EAGLE, MTP, DFlash, and DSpark cover the use case. The [v0.33.0 release notes](https://github.com/vllm-project/vllm/releases/tag/v0.33.0rc3) list it under breaking changes.

Those notes sit on the tag `v0.33.0rc3`, published November 2, 2026, while the bare `v0.33.0` tag page has no notes. A config that still sets `method="medusa"` now fails validation.

The newest vLLM tag on November 8, 2026 is still `v0.33.0`, and `main` does not bring either method back. The [vLLM, SGLang, or llama.cpp comparison](https://aifordevs.org/articles/vllm-sglang-llama-cpp) covers the engines more broadly.

## When speculative decoding hurts

Speculative decoding hurts when the GPU has no compute to spare, when acceptance is low, or when the draft costs too much. vLLM's own 2024 benchmarks showed 1.4x and 1.8x slowdowns on Llama 3 70B at high QPS, and its docs now aim the feature at medium-to-low QPS, memory-bound workloads.

### Load

vLLM's [2024 post](https://vllm.ai/blog/2024-10-17-spec-decode) reported up to 1.5x on ShareGPT at QPS 1 and the slowdowns above at high QPS, on four H100s, with no independent measurement. Verification multiplies the effective batch: vLLM's [dynamic speculative decoding docs](https://docs.vllm.ai/en/latest/features/speculative_decoding/dynamic_speculative_decoding/) say that once batch size times K passes a critical point, decode speed drops. Their fix is a per-concurrency K, with `[[1, 64, 3], [65, 128, 1], [129, 512, 0]]` as the example.

### Low acceptance

High [temperature](https://aifordevs.org/glossary#temperature) and open-ended prose are harder to predict. At an alpha of 0.3 and k = 4, Equation 1 gives 1.43 tokens per pass, and a draft at c = 0.10 brings the speedup to 1.02x. Most draft work is thrown away at that rate.

### Memory for the draft

A separate draft adds its own weights and its own KV cache, which vLLM's memory manager [now tracks alongside the target's](https://vllm.ai/blog/2024-10-17-spec-decode). MTP and EAGLE heads avoid a whole second model, and lookup methods add no weights.

### Lossy verification

Some speedups give up the distribution guarantee on purpose. vLLM's [FLy verification](https://docs.vllm.ai/en/latest/features/speculative_decoding/fly/) defers some rejections and is documented as intentionally lossy, with [1.04x to 1.31x decode throughput reported on MI355X and B300](https://github.com/vllm-project/vllm/releases/tag/v0.33.0rc3) in vLLM's own release notes. Use it only if you test output quality on your task.

## How to measure speculative decoding on your traffic

Measure acceptance, per-token latency, and the batch sizes you actually run. vLLM exposes `vllm:spec_decode_num_draft_tokens` and `vllm:spec_decode_num_accepted_tokens` at `/metrics`, and their ratio is your acceptance rate. Compare time per output token with speculation on and off at your real concurrency.

The metric names come from [vLLM's source](https://github.com/vllm-project/vllm/blob/main/vllm/v1/spec_decode/metrics.py). The per-request flag [`--per-request-spec-decode-metrics summary`](https://docs.vllm.ai/en/latest/features/speculative_decoding/acceptance_metrics/) reports a `mean_acceptance_length` under `metrics.speculative_decoding`, which is experimental and may change between releases.

Run the [speedup function](#how-to-estimate-speculative-decoding-speedup) with your measured `alpha` and `c` before you pay for a new draft model. Watch [throughput](https://aifordevs.org/glossary#throughput) as well as [time to first token](https://aifordevs.org/glossary#ttft), because a high-QPS endpoint can lose throughput while a low-load test still looks fine.

## How to decide on speculative decoding

Turn it on for latency-sensitive, low-concurrency endpoints first, such as interactive chat and code completion. Leave high-throughput batch jobs alone until a load test shows a gain. Use this order when you pick a method, and retest after each engine upgrade.

1. If your model ships MTP heads or an EAGLE checkpoint, start there, because the draft is trained on the target and adds little memory.
2. If your prompts repeat in the output, enable `ngram` or `suffix`, because they cost almost no compute.
3. If neither applies, try a small draft from the same model family and measure `alpha`, `c`, and memory.
4. If you expect high concurrency, set a per-concurrency K of zero at the top of the range.

The [KV cache](https://aifordevs.org/glossary#kv-cache) budget and [batching](https://aifordevs.org/glossary#continuous-batching) strategy sit underneath this choice, and a sparse [mixture-of-experts model](https://aifordevs.org/articles/mixture-of-experts) changes that math again. The [inference topic page](https://aifordevs.org/topics/inference) collects the related guides.

## Questions

### What is speculative decoding in LLMs?

Speculative decoding is a method for generating tokens from a large model faster. A cheap draft proposes several tokens, the large target model checks them all in one forward pass, and every accepted token saves one target step. With the standard acceptance rule, the output distribution is the same as sampling from the target model alone.

### Does speculative decoding change the model's answers?

With the standard acceptance rule from Leviathan et al. and Chen et al., no. The output follows the same distribution as the target alone, apart from hardware numerics. Lossy methods such as FLy, which vLLM documents as intentionally lossy, relax that guarantee for speed and may change outputs. Check which verification policy your config uses.

### What is a good acceptance rate for speculative decoding?

Measure it on your own traffic, because acceptance varies with the draft and the task. The EAGLE paper reports about 0.8 for its method and about 0.6 for Medusa on its benchmark. At an alpha of 0.8 and four draft tokens, Equation 1 gives 3.36 tokens per target pass, and low acceptance near 0.3 gives only about 1.4.

### Why does speculative decoding hurt at high load?

Verification multiplies the effective batch size, because each sequence checks K tokens per step. vLLM's docs say that once batch size times K passes a critical point, decode speed drops. vLLM measured slowdowns of 1.4x and 1.8x on Llama 3 70B at high QPS, so throughput-focused deployments often need a smaller K or none.

### Is Medusa still supported in vLLM?

No. vLLM removed Medusa and the MLP speculator in pull request 61701, merged October 15, 2026, and listed the removal under breaking changes in the v0.33.0 notes published November 2, 2026. A config that still sets method to medusa now fails validation. As of November 8, 2026, no newer vLLM release has reversed that.

### What is the difference between speculative decoding and speculative sampling?

They describe the same family of methods. Leviathan et al. used the term speculative decoding, and Chen et al. used speculative sampling for the same idea, posted independently a few months apart. Both pair a cheap draft with a target model and use modified rejection sampling to keep the target's output distribution.

## Sources

1. [Leviathan et al., Fast Inference from Transformers via Speculative Decoding (2022)](https://arxiv.org/abs/2211.17192)
2. [Chen et al., Accelerating Large Language Model Decoding with Speculative Sampling (2023)](https://arxiv.org/abs/2302.01318)
3. [Cai et al., Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads (2024)](https://arxiv.org/abs/2401.10774)
4. [Li et al., EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (2024)](https://arxiv.org/abs/2401.15077)
5. [Chen, Liang, and Liu, DFlash: Block Diffusion for Flash Speculative Decoding (2026)](https://arxiv.org/abs/2602.06036)
6. [DeepSeek-AI, DeepSeek-V3 Technical Report (2024), Section 2.2 on multi-token prediction](https://arxiv.org/abs/2412.19437)
7. [vLLM v0.33.0 release notes (attached to tag v0.33.0rc3)](https://github.com/vllm-project/vllm/releases/tag/v0.33.0rc3)
8. [vLLM pull request 61701, Remove Medusa and MLP speculator support](https://github.com/vllm-project/vllm/pull/61701)
9. [vLLM speculative configuration on main (November 2026)](https://github.com/vllm-project/vllm/blob/main/vllm/config/speculative.py)
10. [vLLM docs, Speculative Decoding](https://docs.vllm.ai/en/latest/features/speculative_decoding/)
11. [vLLM docs, Multi-Token Prediction](https://docs.vllm.ai/en/latest/features/speculative_decoding/mtp/)
12. [vLLM docs, Dynamic Speculative Decoding](https://docs.vllm.ai/en/latest/features/speculative_decoding/dynamic_speculative_decoding/)
13. [vLLM docs, FLy Verification](https://docs.vllm.ai/en/latest/features/speculative_decoding/fly/)
14. [vLLM docs, Per-Request Acceptance Metrics](https://docs.vllm.ai/en/latest/features/speculative_decoding/acceptance_metrics/)
15. [vLLM metrics source (main, November 2026)](https://github.com/vllm-project/vllm/blob/main/vllm/v1/spec_decode/metrics.py)
16. [vLLM blog, How Speculative Decoding Boosts vLLM Performance (2024)](https://vllm.ai/blog/2024-10-17-spec-decode)
17. [SGLang docs, Speculative Decoding](https://docs.sglang.io/docs/advanced_features/speculative_decoding)

## Cite this deep dive

AI for Devs. "Speculative decoding: how drafts speed up LLM inference." Published November 6, 2026, updated November 8, 2026. https://aifordevs.org/articles/speculative-decoding

- Page: https://aifordevs.org/articles/speculative-decoding
- Markdown: https://aifordevs.org/articles/speculative-decoding.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs
- Published: 2026-11-06
- Updated: 2026-11-08