Prefill vs Decode: The Two Halves of Every LLM Request

When an ML engineer looks at a Large Language Model (LLM) request, they often see a single black box: "Input prompt -> Output text." This view obscures the fundamental architectural split that dictates hardware selection, cost, and performance. LLM inference is composed of two distinct computational phases: Prefill and Decode.

This article is Part 3 of the LLM Inference from Scratch series.

Understanding these phases is critical because they impose opposite constraints on hardware. Prefill is compute-bound, requiring massive floating-point throughput. Decode is memory-bound, requiring high memory bandwidth. Confusing these leads to poor infrastructure choices—such as buying a GPU with high TFLOPS but low memory bandwidth for a system designed for high-concurrency streaming.

The Mechanics of State: Context vs. Generation

To understand the split, we must look at the Transformer architecture’s forward pass. The core operation is the Matrix Multiplication within the attention mechanism and feed-forward networks.

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

Here, $Q$ (Query), $K$ (Key), and $V$ (Value) are derived from the input embeddings. The critical variable is the sequence length ($S$), which represents the number of tokens being processed.

1. Prefill: Parallel Processing of the Prompt

During the prefill phase, the model processes the entire user prompt (and any existing conversation history) in a single step. If a user sends a prompt with 100 tokens, the GPU computes the output for all 100 tokens simultaneously.

This is a parallel operation. The hardware launches thousands of threads to compute the matrix multiplications for all tokens in the batch and sequence length at once.

• Mechanism: The input tensor shape is $(B, S, D)$, where $B$ is batch size, $S$ is sequence length, and $D$ is hidden dimension. The GPU computes the attention scores and output vectors for all $S$ tokens in parallel.
• Hardware Demand: Since the calculation volume is huge relative to the data movement, this phase is compute-bound. It saturates the GPU's Tensor Cores (ALUs).
• Primary Metric: Latency. Specifically, Time to First Token (TTFT). Users perceive the wait time until the first word appears as the total response time.

2. Decode: Sequential Generation

Once the prefill phase completes, the model enters the decode phase. Here, the model generates tokens one by one.

If the prompt had 100 tokens, the prefill phase produced the 101st token. Now, the model takes that 101st token, appends it to the context, and generates the 102nd token. Then it takes the 102nd token, appends it, and generates the 103rd.

• Mechanism: At step $t$, the input sequence length is $S + t$. The model computes attention for only one new token, but it must attend to all previous tokens ($S + t - 1$).
• Hardware Demand: As generation progresses, the sequence length grows. The amount of computation per token remains roughly constant (one token forward pass), but the data movement increases. The model must fetch the entire KV Cache (Key-Value cache) from memory for every single token generated.
• Primary Metric: Throughput (tokens/second). The bottleneck is how fast memory can be read, not how fast the ALUs can calculate. This makes decode memory-bound. The memory bandwidth limits the maximum throughput (tokens/sec) for a given batch, which directly impacts inter-token latency.

Why Prefill is Compute-Bound

In the prefill phase, the GPU is performing a massive amount of arithmetic. For a prompt of length 1024, the attention mechanism computes a $1024 \times 1024$ attention matrix. This involves billions of multiply-add operations.

The data (weights and activations) is loaded from HBM (High Bandwidth Memory) into the GPU's on-chip SRAM/cache. Once loaded, the data is reused many times for computation. The ratio of computation to memory access is high. Therefore, the speed is limited by how many operations the GPU can perform per second (FLOPS).

Example: A NVIDIA A100 with 1,935 GB/s memory bandwidth and 312 TFLOPS (FP16 with sparsity) will process a long prompt quickly because its compute cores are saturated. Even if you increase memory bandwidth, the prefill speed won't improve significantly because the compute cores are already the bottleneck.

Why Decode is Memory-Bound

In the decode phase, the scenario flips. For each new token, the model must:

1. Fetch the model weights (constant size, e.g., 7B parameters = ~14GB for FP16).
2. Fetch the KV Cache, which grows with every token generated.

The KV Cache stores the $K$ and $V$ matrices for all previous tokens. For a 7B parameter model with FP16 precision, each layer's KV cache per token is approximately 16KB (assuming 4096 hidden dim, 32 heads, 128 head dim: $2 \times 32 \times 128 \times 2$ bytes $\approx$ 16KB per token).

If you generate 1000 tokens, the KV Cache grows significantly. For the next token, the GPU must read this entire block from memory, plus the model weights, to perform a tiny amount of computation (one token's forward pass).

The computation volume is small compared to the data movement. The GPU's ALUs sit idle, waiting for memory to deliver data. This is the definition of a memory-bound workload. The speed is limited by memory bandwidth (GB/s), not compute throughput.

Implication: To improve decode speed, you need more memory bandwidth, not more compute cores. This is why GPUs with higher memory bandwidth (e.g., H100 with 3TB/s vs. A100 with 1,935 GB/s) show dramatic improvements in decode latency, even if their compute specs are similar.

Key Metrics: TTFT vs. Throughput

Understanding the prefill/decode split clarifies the primary performance metrics:

• TTFT (Time to First Token): Dominated by prefill latency. A long prompt increases prefill time linearly. Users care about this for "chat" experiences where responsiveness is key.
• Decoding Latency (Inter-Token Latency): The time between generating one token and the next. Dominated by decode memory bandwidth. This affects the smoothness of streaming.
• Throughput (Tokens/Second): Total tokens generated per second across all requests. Dominated by decode efficiency. High throughput requires maximizing the number of concurrent decode batches, which is limited by KV Cache memory capacity.

Optimization Strategies

Because the phases have different bottlenecks, optimization strategies differ:

For Prefill (Compute-Bound):

• FlashAttention: Optimizes the attention mechanism to reduce global memory bandwidth pressure by using tiling and recomputation, allowing the GPU to sustain higher effective compute throughput on long sequences where memory might otherwise stall the pipeline.
• Parallelism: Use tensor parallelism to split large matrices across multiple GPUs, increasing effective compute throughput.

For Decode (Memory-Bound):

• KV Cache Quantization: Store KV Cache in INT8 or INT4 instead of FP16. This reduces memory footprint, allowing more batches to fit in memory and reducing bandwidth pressure.
• PagedAttention: (Used in vLLM) Manages KV Cache memory efficiently, preventing fragmentation and allowing higher batch sizes.
• Speculative Decoding: Use a small, fast model to draft multiple tokens, then verify them with the large model. This reduces the number of full forward passes required, effectively hiding memory latency.

Conclusion

LLM inference is not a monolithic process. The prefill phase is a compute-heavy parallel operation that determines TTFT. The decode phase is a memory-heavy sequential operation that determines throughput and inter-token latency.

For backend engineers, this means:

1. Optimize for the bottleneck: Use compute-optimized GPUs for prefill-heavy workloads (e.g., long document analysis). Use memory-bandwidth-optimized GPUs for decode-heavy workloads (e.g., high-concurrency chat).
2. Monitor separately: Track TTFT and tokens/second independently. A drop in TTFT might indicate better prefill optimization, while a drop in tokens/second indicates KV Cache or memory bandwidth issues.

Ignoring this distinction leads to misaligned hardware investments and suboptimal system design. The future of LLM inference lies in architectures that can dynamically balance these two halves, but for now, understanding the split is the first step to mastering performance.

FAQ

What is the difference between TTFT and Inter-Token Latency? TTFT (Time to First Token) measures the delay before the model starts speaking, driven by the compute-heavy prefill phase. Inter-Token Latency is the time between subsequent words, driven by the memory-bound decode phase.

Why is Decode memory-bound? In decode, the model generates one token at a time but must read the entire growing KV Cache from memory for each step. The computation is small relative to the data movement, making memory bandwidth the limiting factor.

How does KV Cache size affect performance? As the sequence length grows, the KV Cache grows linearly. A larger cache increases the amount of data that must be fetched from memory for every single token generated, slowing down decode throughput and increasing inter-token latency.

Common Pitfalls

• Assuming higher TFLOPS always improves decode speed: High compute throughput helps prefill, but decode is limited by memory bandwidth. A faster GPU won't help if it lacks bandwidth.
• Ignoring KV Cache fragmentation: Without efficient memory management (like PagedAttention), KV Cache fragmentation can prevent large batches from fitting in memory, reducing throughput.
• Confusing prefill and decode bottlenecks: Optimizing for one phase (e.g., reducing prefill latency) may not help the other (decode throughput) and vice versa. They require different hardware and software strategies.

Practical Takeaways

• Prefill is compute-bound, Decode is memory-bound: Treat them as distinct workloads with different hardware requirements.
• Memory bandwidth is king for high-concurrency chat: For systems serving many users generating long responses, prioritize GPUs with high GB/s bandwidth.
• Monitor TTFT and Throughput separately: Use these metrics to diagnose whether your bottleneck is in the initial processing (prefill) or the ongoing generation (decode).