Speculative Decoding: 2x Throughput for Free?

Large language model (LLM) inference is typically bound by memory bandwidth, not compute. In standard autoregressive decoding, the model generates one token at a time. For each token, the entire context window must be processed to compute the next probability distribution. This serialization means that doubling the sequence length does not double the latency, but it does linearly increase the time between tokens (TTFT and Inter-Token Latency). Speculative decoding (SD) attempts to break this serialization bottleneck by allowing the system to predict multiple tokens ahead, verify them in a single batched pass, and discard invalid ones.

The Bottleneck: Serialized Autoregression

Consider a standard LLM inference loop. The input is a sequence of tokens $X = [x_1, x_2, ..., x_n]$. To generate $x_{n+1}$, the model computes:

$P(x_{n+1} | X) = \text{Softmax}(W_{out} \cdot \text{Transformer}(X))$

This computation is performed sequentially. Even if the GPU is idle after processing $x_{n+1}$, it cannot start computing $x_{n+2}$ until $x_{n+1}$ is fully generated. This creates a "latency wall" where the throughput (tokens per second) is capped by the time required to process one token through the full transformer layers.

The Speculative Hypothesis

Speculative decoding introduces a second, smaller model—the draft model—to hypothesize the next $k$ tokens. The draft model is typically a smaller version of the target LLM (e.g., a 7B parameter model drafting for a 70B parameter model). Because the draft model is smaller, it generates tokens significantly faster than the target model.

The core insight is that the output distribution of a small model often closely approximates that of a large model. This approximation holds because smaller models trained on the same data share similar linguistic patterns and structural priors, even if they lack the capacity for rare or complex reasoning. If the draft model can accurately predict the next few tokens, the large model does not need to compute them from scratch. Instead, it only needs to verify them against its more precise distribution.

The Mechanism: Draft and Verify

The speculative decoding process operates in a loop with two distinct phases:

1. Drafting Phase

The draft model generates $k$ tokens autoregressively. Let the draft model be $M_d$ and the target model be $M_t$. Given the current context $X$, $M_d$ generates a sequence of $k$ draft tokens:

$D = [d_1, d_2, ..., d_k]$

where $d_i$ is the $i$-th draft token. This phase is still sequential but much faster because $M_d$ has fewer parameters and less compute per layer.

2. Verification Phase

The target model $M_t$ takes the original context $X$ concatenated with the draft sequence $D$ as input. Crucially, $M_t$ performs a single forward pass on the combined sequence $[X, D]$. While this processes all $k$ draft tokens simultaneously in terms of kernel execution, it is not "parallel" in the sense of independent computation. The causal attention mask ensures that the prediction for each token $d_i$ depends strictly on the previous tokens $X, d_1, ..., d_{i-1}$. This is a batched computation that amortizes the cost of the target model's heavy layers across multiple tokens.

For each draft token $d_i$, $M_t$ computes its own probability distribution $P_t(d_i | X, d_1, ..., d_{i-1})$. The verification uses a rejection sampling criterion to decide whether to accept or reject each draft token.

Acceptance Criteria and Rejection Sampling

The acceptance criterion, first formalized by Leviathan et al. in their 2023 paper "Fast Inference from Transformers via Speculative Decoding," ensures that the final output distribution remains identical to what would have been produced by sampling from $M_t$ alone. This is achieved through a Metropolis-Hastings correction mechanism. The formula $\min(1, P_t/P_d)$ is a simplified representation of this correction, which adjusts for discrepancies between the draft and target distributions to maintain statistical correctness.

For each draft token $d_i$, let $P_t(d_i)$ be the probability assigned by the target model, and $P_d(d_i)$ be the probability assigned by the draft model. The token $d_i$ is accepted with probability:

$A = \min\left(1, \frac{P_t(d_i)}{P_d(d_i)}\right)$

If the token is accepted, it is added to the output sequence. If rejected, the target model samples a new token from its own distribution, and the speculative chain breaks. Any subsequent draft tokens are discarded.

Example Scenario

Imagine a draft model predicts the next token is "cat" with probability 0.8. The target model assigns "cat" a probability of 0.9. The acceptance probability is $\min(1, 0.9/0.8) = 1$. The token "cat" is accepted.

Now, imagine the draft model predicts "dog" with probability 0.5, but the target model assigns "dog" only 0.3. The acceptance probability is $\min(1, 0.3/0.5) = 0.6$. The system rolls a random number; if it is less than 0.6, "dog" is accepted. If not, "dog" is rejected, and the target model samples a replacement token (say, "bird"). The speculative sequence terminates at "bird".

Throughput Analysis: Why 2x?

The theoretical speedup of speculative decoding depends on the acceptance rate ($\alpha$) and the number of draft tokens ($k$).

Let $T_d$ be the time to generate one draft token with the small model, and $T_t$ be the time to generate one token with the large model. In standard decoding, generating $k$ tokens takes $k \cdot T_t$.

In speculative decoding with $k$ draft tokens:

1. Drafting takes $k \cdot T_d$.
2. Verification takes $T_t$ (batched forward pass).
3. If all $k$ tokens are accepted, the total time is $k \cdot T_d + T_t$.

The speedup $S$ is:

$S = \frac{k \cdot T_t}{k \cdot T_d + T_t} = \frac{k}{k \cdot \frac{T_d}{T_t} + 1}$

If $T_d \ll T_t$ (the draft model is much faster) and $\alpha \approx 1$ (high acceptance rate), the speedup approaches $k$. In practice, $T_d$ is not zero, and acceptance rates drop as the draft deviates from the target. However, even with moderate acceptance rates, using $k=4$ or $k=8$ draft tokens can yield 1.5x to 2x throughput improvements.

Tradeoffs and Limitations

Speculative decoding is not free. It introduces several overheads that must be balanced against the throughput gains:

1. Compute Cost of Verification: The primary computational cost is the target model's forward pass on the extended sequence. While this is amortized over $k$ tokens, it is still the dominant expense. If the draft model is not significantly faster, the drafting phase becomes a bottleneck rather than a help.
2. Memory Management Overhead: Managing the KV cache for the draft tokens during verification increases memory usage. This is distinct from the computational cost; it involves allocating memory for the draft tokens' keys and values and potentially causing cache misses if the sequence length grows too large or if the memory footprint exceeds the GPU's capacity.
3. Compute Imbalance: If the draft model architecture is similar to the target (e.g., same attention heads but fewer layers), the drafting phase may not be fast enough to offset the verification cost. The ideal draft model is one that can generate tokens in a fraction of the time of the target model.
4. Variability: The variable number of accepted tokens leads to non-deterministic latency. While average throughput improves, tail latency may worsen if many drafts are rejected, causing pipeline stalls.

Conclusion

Speculative decoding achieves throughput improvements by shifting the computational load from the expensive target model to a cheaper draft model, while leveraging a single batched forward pass to amortize the cost of the target model's verification. The 2x speedup is not guaranteed; it is a function of the draft model's accuracy (acceptance rate) and the relative speed difference between draft and target models. For ML engineers, optimizing the draft model selection and tuning the number of draft tokens $k$ are critical levers for maximizing inference efficiency.

Common Pitfalls

1. Ignoring Memory Limits: Selecting a draft model that is too large or setting $k$ too high can exhaust GPU memory, leading to OOM errors or swapping, which destroys performance.
2. Mismatched Architectures: Using a draft model with a fundamentally different architecture (e.g., RNN vs. Transformer) can lead to poor acceptance rates because the latent spaces do not align well.
3. Overlooking Tail Latency: Optimizing only for average throughput can hide spikes in latency caused by frequent rejections, which is detrimental for interactive applications requiring consistent response times.

Practical Takeaways

1. Draft Model Selection: Prioritize draft models that are significantly smaller and faster than the target, even if their raw accuracy is slightly lower, as long as the acceptance rate remains high.
2. Dynamic $k$ Tuning: Consider implementing dynamic draft lengths that adjust $k$ based on the current acceptance rate, increasing $k$ when the draft is confident and decreasing it when confidence drops.
3. Benchmarking: Always measure both average throughput and tail latency (p99) in your specific hardware environment, as theoretical speedups rarely match real-world performance due to system overheads.

FAQ

Q: Does speculative decoding affect the quality of the generated text? A: No. The acceptance criteria ensure that the final output distribution is mathematically identical to sampling directly from the target model. The quality remains unchanged.

Q: Can I use any small model as a draft? A: Ideally, the draft model should be trained on the same data as the target model. Models fine-tuned or distilled from the target usually yield higher acceptance rates.

Q: Is speculative decoding suitable for very short sequences? A: It is most effective for longer sequences where the overhead of the draft model is amortized over many tokens. For very short sequences, the overhead may outweigh the benefits.