Medusa, EAGLE, and Lookahead Decoding Compared

Large Language Models (LLMs) are bottlenecked by autoregressive decoding: the model must generate one token at a time, waiting for the previous token's embedding before computing the next. This sequential dependency creates a latency floor proportional to the sequence length. Speculative decoding breaks this bottleneck by using a "draft" mechanism to propose multiple tokens ahead of time, which are then verified in parallel by the large model. If the verification succeeds, the batch is accepted; if not, the large model corrects the path.

Medusa, EAGLE, and Lookahead Decoding represent three distinct architectural approaches to this problem. While they share the same high-level goal, their mechanisms for generating drafts, training objectives, and verification logic differ significantly.

The Baseline: Speculative Decoding

In standard speculative decoding, a small, fast draft model (e.g., TinyLlama) generates $k$ candidate tokens based on the current context. The large model then processes these $k$ tokens in parallel to compute their probabilities. The draft tokens are accepted if they match the large model's top-$k$ predictions. If a mismatch occurs, the large model generates a new token from that point, and the process repeats.

The efficiency gain comes from the large model performing $k$ forward passes in parallel rather than sequentially. However, the success of this approach depends entirely on how well the draft model predicts the large model's output.

Medusa: Parallel Prediction Heads

Medusa (Multiple Decoding Heads) takes a unique approach by avoiding a separate draft model altogether. Instead, it adds shallow, parallel decoding heads on top of the frozen LLM's transformer layers.

Mechanism

During inference, the LLM computes its hidden states as usual. Medusa attaches $m$ additional linear heads (or shallow networks) to these hidden states. Each head is responsible for predicting a specific future token position: Head 1 predicts $t+1$, Head 2 predicts $t+2$, and so on.

Crucially, these heads operate in parallel. They all receive the same hidden state from the current step $t$ and independently predict the next $m$ tokens. The large model then verifies these $m$ tokens by processing the draft sequence through the base model, leveraging KV-cache reuse to avoid redundant computation of the initial context.

# Conceptual representation of Medusa's parallel heads
hidden_state = base_llm(input)
token_t1 = head_1(hidden_state)
token_t2 = head_2(hidden_state)
token_t3 = head_3(hidden_state)
# All predictions are verified against the base LLM in one go

Training

Medusa is trained by freezing the base LLM and adding these heads. The loss function is a standard cross-entropy loss for each head, predicting the next token based on the current hidden state. This allows Medusa to leverage the rich semantic information already encoded in the LLM's layers without training a separate architecture.

Trade-offs

• Memory Overhead: Adding multiple heads increases the model's parameter count slightly, but the main cost is the memory required to store the heads' weights and activations.
• Simplicity: Integration is straightforward—just add heads to the existing model.
• Limitation: Since each head predicts independently from the same state, Medusa does not account for the sequential dependency between predicted tokens (i.e., the prediction of $t+2$ does not use the prediction of $t+1$).

EAGLE: Recursive Small Model with Eagle Loss

EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency) uses a separate, smaller draft model but introduces a critical innovation in how it is trained: predicting at the feature level rather than the token level.

Mechanism

EAGLE employs a small, lightweight draft model that generates tokens recursively. It predicts $t+1$, then uses that prediction to predict $t+2$, and so on. This recursive structure mirrors the autoregressive nature of the large model, making it more aligned with the actual generation process.

Feature-Level Drafting

The key insight of EAGLE is that predicting directly at the token level is intrinsically uncertain, since sampling introduces randomness that a small model cannot easily anticipate. Instead, EAGLE's draft model operates on the large model's second-to-top-layer feature (hidden state) representations, which are more predictable and easier to extrapolate than the sampled tokens themselves.

The draft model is trained with a combined objective: a regression loss comparing predicted and true hidden-state features, and a classification (cross-entropy) loss on the resulting token, computed by passing the predicted feature through the frozen LM head. This combined objective encourages the draft model to produce features that lead to accurate token predictions across the recursive chain.

Trade-offs

• Training Complexity: Requires training a separate small model with a specialized loss function.
• Memory Efficiency: Uses less memory than Medusa's multiple heads, as it only involves one small model.
• Alignment: The recursive structure better matches the large model's behavior, leading to higher acceptance rates.

Lookahead Decoding: Contextual and Structural Drafting

Lookahead Decoding, as defined in "Break the Sequential Dependency of LLM Inference Using Lookahead Decoding" by Fu et al., generally refers to methods that use contextual or structural priors to draft tokens. In this context, it specifically denotes the algorithm that combines Jacobi-iteration-based parallel decoding with n-gram pool matching, distinguishing it from purely neural draft approaches.

Mechanism

Lookahead Decoding does not rely on a separate draft model. Instead, it uses Jacobi iteration—a parallel fixed-point decoding method—to generate multiple candidate n-grams for future positions simultaneously, caching them in an n-gram pool. Unlike Medusa's parallel heads or EAGLE's recursive small model, Lookahead Decoding requires no additional trained component. The key distinction lies in the verification strategy: candidate n-grams pulled from the pool are verified directly against the large model's own token-by-token generation.

Some variants of Lookahead Decoding rely purely on n-gram matching against the context window, providing a strong prior based on repeated patterns. Others generate candidates via parallel Jacobi iteration and verify the entire block at once, using a different acceptance criterion.

Comparison with EAGLE

While EAGLE trains a draft model with a combined feature-regression and token-classification objective, Lookahead Decoding requires no training at all, relying instead on algorithmic n-gram generation and matching. This makes Lookahead Decoding easy to deploy without additional training but potentially less adaptable than EAGLE's learned recursive model.

Trade-offs

• Flexibility: N-gram-based Lookahead is highly effective for repetitive or structured text (e.g., code, logs).
• Generalization: May struggle with novel or creative text where n-gram patterns are weak.
• Verification: Block-wise verification can be less efficient than step-wise verification if the acceptance rate is low.

Summary of Differences

Conclusion

Medusa, EAGLE, and Lookahead Decoding offer different paths to accelerating LLM inference. Medusa is the simplest to integrate, leveraging parallel heads for immediate speedups. EAGLE offers higher efficiency through recursive small models and specialized training, making it ideal for production deployments where acceptance rates matter. Lookahead Decoding, particularly with n-gram priors, excels in structured domains. The choice depends on the trade-off between integration complexity, memory constraints, and the nature of the text being generated.