EAGLE: feature-level drafting

I covered speculative decoding earlier in this journey. The core idea is beautiful: use a cheap draft model to propose multiple tokens, then verify them all in parallel with the expensive target model. If the draft model is good enough, you accept most tokens and skip multiple sequential decode steps.

The weakness of standard speculative decoding is that the draft model is a separate, smaller model. It has its own weights, its own distribution, and it inevitably disagrees with the target model on some fraction of tokens. Typical acceptance rates are 60% to 80% depending on the task and the draft/target pair. Every rejection wastes the compute spent on all subsequent draft tokens in that sequence.

EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency) takes a fundamentally different approach, and the results are impressive.

The key insight: predict features, not tokens

Standard speculative decoding drafts in token space: the draft model outputs a probability distribution over the vocabulary, you sample a token, feed it back, and repeat. EAGLE drafts in feature space: it predicts the hidden state (the feature vector at the last transformer layer) that the target model would produce for the next token.

Why does this matter? Because predicting features is easier than predicting tokens. The token distribution is a softmax over a 32K+ vocabulary with complex, peaky distributions. The feature vector is a continuous, relatively smooth representation in a lower-dimensional space. A small network can learn to extrapolate features from the recent feature trajectory with high accuracy.

Architecture of the EAGLE head

EAGLE adds a lightweight autoregressive head on top of the target model. The head takes as input:

• The hidden state from the target model's last layer at position t
• The embedding of the token at position t+1 (from the target model's embedding layer)

And predicts the hidden state at position t+1. This predicted feature is then passed through the target model's existing LM head (the unembedding matrix) to get a token distribution.

# EAGLE drafting (simplified)
class EAGLEHead(nn.Module):
    def __init__(self, hidden_size):
        super().__init__()
        # Single transformer layer as the draft head
        self.fc = nn.Linear(hidden_size * 2, hidden_size)
        self.transformer_layer = TransformerDecoderLayer(hidden_size)

    def forward(self, hidden_state, token_embedding):
        # Concatenate hidden state and token embedding
        x = torch.cat([hidden_state, token_embedding], dim=-1)
        x = self.fc(x)
        x = self.transformer_layer(x)
        return x  # predicted next hidden state

# During drafting:
def eagle_draft(model, eagle_head, hidden_states, num_draft=5):
    draft_tokens = []
    h = hidden_states[-1]  # last hidden state from target model
    token_emb = model.embed(last_token)

    for _ in range(num_draft):
        h_pred = eagle_head(h, token_emb)
        logits = model.lm_head(h_pred)  # reuse target model's LM head
        token = sample(logits)
        draft_tokens.append(token)
        token_emb = model.embed(token)
        h = h_pred
    return draft_tokens

The EAGLE head is tiny: typically a single transformer layer with the same hidden dimension as the target model. For a 7B model with hidden size 4096, the EAGLE head adds roughly 0.24B parameters (about 3% overhead). For a 70B model, the overhead is even smaller proportionally.

Why acceptance rates are higher

The reason EAGLE achieves higher acceptance rates than standard speculative decoding comes down to what the draft head has access to:

• Standard spec decode: The draft model processes the same input tokens as the target but with different weights, different internal representations, and a different learned distribution. It approximates the target from the outside.
• EAGLE: The draft head receives the target model's own hidden states. It is literally looking at the target model's internal computation and predicting what will happen next. It approximates the target from the inside.

The published results show acceptance rates of 75% to 85% on coding tasks and 80% to 90% on conversational tasks, compared to 55% to 75% for standard speculative decoding with a well-matched draft model.

Training the EAGLE head

Training is straightforward and cheap. You freeze the target model entirely and train only the EAGLE head to minimize the L2 distance between predicted and actual hidden states:

# Training objective (simplified)
for batch in dataloader:
    with torch.no_grad():
        # Run target model, collect hidden states
        hidden_states = target_model(batch, output_hidden_states=True)

    # Train EAGLE head to predict next hidden state
    for t in range(seq_len - 1):
        h_pred = eagle_head(hidden_states[t], target_model.embed(batch[t+1]))
        loss = F.mse_loss(h_pred, hidden_states[t+1])
        loss.backward()
        optimizer.step()

Training typically takes 1 to 2 days on a single GPU using a few thousand examples from ShareGPT or similar conversational data. The cost is negligible compared to pretraining or even fine-tuning the target model.

Serving with EAGLE

From a serving perspective, EAGLE is attractive because:

• No separate draft model. You do not need to load, manage, or schedule a second model. The EAGLE head is a small addition to the existing model.
• Shared KV cache. The draft tokens are verified by the target model in the same forward pass. There is no KV cache duplication between draft and target.
• Tunable draft length. You can adjust the number of draft tokens (typically 3 to 8) based on the acceptance rate you observe at runtime.

In practice, EAGLE provides a 1.5x to 2.5x speedup in tokens per second for single-request latency scenarios. The speedup is highest for greedy decoding (temperature=0) where the distribution is peaky and predictions are easier, and lower for high-temperature sampling where the output is more random.

Comparison with Medusa

Medusa is another approach that adds extra heads to the target model, but it works differently. Medusa trains multiple independent heads, each predicting a different future token position (head 1 predicts token t+1, head 2 predicts t+2, etc.). These heads operate in parallel but independently, without the autoregressive chaining that EAGLE uses.

The tradeoff:

• Medusa is simpler and can draft all positions in a single forward pass of the heads.
• EAGLE has higher accuracy because each draft step conditions on the previous predicted feature, capturing sequential dependencies.
• In benchmarks, EAGLE typically achieves 1.5x to 2x higher speedup than Medusa on the same model.

EAGLE's core lesson for inference engineering: the best draft model is not a separate model at all. It is a lightweight predictor that sits inside the target model and leverages its internal representations. The closer the drafter is to the target, the higher the acceptance rate.

We skip a day and then dive into MoE routing from scratch, building the gating mechanism that decides which expert processes each token in a Mixture-of-Experts model.