Disaggregated Prefill & Decode: Optimizing LLM Serving

Disaggregated Prefill and Decode

In large language model (LLM) inference, the workload is not uniform. It consists of two distinct phases with radically different computational profiles: prefill and decode. The prefill phase processes the entire input prompt in parallel, consuming massive amounts of compute but only once per request.

The decode phase generates output tokens one by one, requiring low-latency access to the growing Key-Value (KV) cache but utilizing significantly less compute per step.

Traditional "colocated" serving architectures run both phases on the same GPU cluster. This approach forces a compromise: you either over-provision for the memory-hungry prefill (leading to idle decode capacity) or under-provision for the latency-sensitive decode (causing bottlenecks). Disaggregated serving solves this by separating these phases into distinct clusters or even distinct hardware types, connected by a high-speed network. This separation allows each phase to be scaled independently and optimized for its specific mechanism.

This article is Part 4 of the "Distributed & MoE Serving" series.

The Monolithic Bottleneck

To understand the value of disaggregation, we must first look at the failure mode of colocated systems. In a standard colocated setup, a request arrives at a scheduler. The scheduler assigns the entire request—prompt plus generation—to a single GPU or a tensor-parallel group.

During the prefill phase, the GPU computes attention for all input tokens simultaneously. This is highly parallelizable and efficient. However, it requires loading the entire prompt’s KV cache into HBM (High Bandwidth Memory). For long prompts (e.g., 32k+ tokens), this memory footprint is substantial.

Once prefill completes, the system enters the decode phase. Here, the GPU computes one new token at a time. Crucially, it must read the entire KV cache from memory for every single token generated. As the context grows, memory bandwidth becomes the bottleneck, not compute. The GPU sits idle waiting for data fetches.

The problem arises when requests mix. A batch containing one 32k-token prompt and ten 10-token prompts will see the GPU saturated by the long prompt during prefill, while the short prompts sit in the queue. Once the long prompt moves to decode, the GPU’s memory bandwidth is consumed by that single request, potentially throttling the other requests if they share the same resource pool. More importantly, the hardware requirements diverge: prefill benefits from high compute density (FP8/INT8 matmul), while decode benefits from high memory bandwidth and low latency. Forcing them onto the same silicon leads to poor utilization rates. This inefficiency is precisely why pd disaggregation has emerged as a critical architectural pattern for high-throughput production systems.

Mechanism of Separation

Disaggregated serving decouples these phases. You now have two distinct worker pools:

1. Prefill Nodes: Optimized for high compute throughput. They can use tensor parallelism across many GPUs to process long prompts quickly. They do not need to retain state after the initial forward pass.
2. Decode Nodes: Optimized for memory bandwidth and low-latency inference. They may use fewer GPUs with higher memory capacity or specialized accelerators designed for streaming inference.

The interaction happens via the kv-cache. When a prefill node finishes processing the input prompt, it does not generate tokens. Instead, it serializes the resulting KV cache states and sends them over the network to a decode node. The decode node then begins generating output tokens, reading its local KV cache for each step.

This separation allows you to scale prefill nodes based on input traffic volume and decode nodes based on output generation duration. If your users send many long prompts but generate short responses, you scale up prefill and down decode. If you have many chatbots generating long continuations, you scale decode.

Hardware and Performance Benchmarks

Industry benchmarks and papers from projects like vLLM and Megatron-LM highlight the divergence in hardware needs. Prefill operations are dominated by GEMM (General Matrix Multiply) operations, where compute-bound GPUs like the NVIDIA H100 excel due to their high TFLOPS. In contrast, decode operations are memory-bound, often limited by HBM bandwidth (e.g., ~3TB/s on H100). Research indicates that while colocated systems waste up to 40% of compute cycles during decode due to memory stalls, disaggregated systems can achieve near-linear scaling for prefill workloads by dedicating resources solely to matrix multiplication throughput.

KV Cache Transfer Strategies

The critical engineering challenge in disaggregated serving is the transfer of the KV cache from prefill to decode. This transfer must happen before decode begins, but it cannot stall the system. There are two primary mechanisms for this transfer:

1. Full KV Cache Transfer

In this model, the prefill node computes the entire KV cache for the input prompt and sends it as a single block to the decode node. This is simple but network-intensive. For a 32k-token prompt with a model size of 7B parameters, the KV cache might occupy several gigabytes of memory. Transferring this over a 100Gbps network takes time, introducing latency before the first token is generated (Time to First Token, or TTFT).

However, because the decode node has the full cache upfront, it can operate without further network intervention during generation. This is suitable for scenarios where TTFT is acceptable and generation length is moderate.

2. Incremental / Streaming Transfer

For very long contexts or ultra-low TTFT requirements, full transfer is prohibitive. Instead, the prefill node streams chunks of the KV cache to the decode node as they are computed. The decode node begins generating tokens for earlier parts of the context while the prefill node is still computing later parts.

This requires careful synchronization. The decode node must manage a sliding window or a distributed KV cache structure. If the network lags, the decode node may stall. Therefore, this approach demands extremely low-latency networks, such as InfiniBand or RoCE (RDMA over Converged Ethernet), to ensure the data pipeline remains full.

3. Hybrid: KV Cache Offloading

Some architectures allow the KV cache to reside partially on the prefill node and partially on the decode node. The prefill node computes the initial tokens and keeps the corresponding KV entries. As the decode progresses, the KV cache for older tokens is offloaded to the decode node’s memory or even to CPU/Disk storage if GPU memory is exhausted. This reduces the initial transfer size but adds complexity to the attention mechanism, which must now query across multiple memory locations.

Infrastructure Implications

Implementing disaggregated serving is not just a software change; it is an infrastructure shift.

Network Topology: The link between prefill and decode nodes must be high-bandwidth and low-latency. If the network is slow, the decode node will starve, and the prefill node will become a bottleneck due to backpressure. This often necessitates dedicated network fabrics separate from the control plane or training traffic.

Scheduling Complexity: The scheduler now has to track two states: pending prefill tasks and active decode tasks. It must match incoming prompts to available prefill capacity and then route the resulting KV cache to an available decode slot. This requires a more sophisticated orchestrator than standard Kubernetes job queues. Tools like Ray Serve or custom schedulers built on top of vLLM are often required to manage this state machine.

Hardware Heterogeneity: You can now buy different hardware. Prefill nodes might be equipped with A100s or H100s for their compute power. Decode nodes might use L40S or even specialized inference chips like Groq or Cerebras, which are optimized for sequential token generation and have different memory hierarchies. This flexibility can reduce costs significantly compared to a homogeneous cluster.

Common Pitfalls

While powerful, disaggregation introduces several pitfalls that engineers must mitigate:

• Network Saturation: If the prefill-to-decode link is congested, TTFT spikes dramatically. Monitoring network utilization between clusters is critical.
• Serialization Overhead: Serializing and deserializing the KV cache at scale consumes CPU cycles and adds latency. Efficient serialization formats (like FlatBuffers or Protobufs) are essential.
• Scheduling Latency: Poorly tuned schedulers can introduce delays in matching prefill outputs to decode slots, effectively negating the benefits of separation.
• Cache Eviction Mismanagement: In hybrid models, premature eviction of KV cache entries can lead to recomputation penalties or degraded response quality.

Practical Takeaways

When implementing disaggregated serving, consider these key engineering decisions:

1. Choose Transfer Strategy Based on Use Case: Use Full KV Cache Transfer for short-to-medium prompts where simplicity and zero-network-intervention during generation are priorities. Opt for Incremental/Streaming Transfer only if you have ultra-low latency requirements and robust RDMA-capable infrastructure.
2. Invest in Network Fabric: The performance of your disaggregated system is capped by the network between prefill and decode nodes. Do not skimp on bandwidth or latency here.
3. Heterogeneous Hardware is Key: Leverage the separation to buy the right tool for the job. Use compute-dense GPUs for prefill and memory-bw-optimized or specialized chips for decode.
4. Monitor TTFT and Decode Throughput Separately: Track these metrics independently to identify whether bottlenecks lie in the prefill cluster, the network, or the decode cluster.

Conclusion

Disaggregated serving is not a silver bullet. It introduces network dependency and scheduling complexity. However, for production workloads with variable input lengths and high throughput requirements, the efficiency gains are substantial. By separating the compute-heavy prefill from the memory-bound decode, organizations can optimize each phase independently, reduce wasted GPU cycles, and better align infrastructure costs with actual usage patterns. As LLMs grow larger and context windows longer, the divergence between these phases will only widen, making disaggregation an increasingly necessary architectural pattern.

FAQ

Is disaggregated serving worth the complexity? For high-throughput production environments with mixed prompt lengths, yes. The efficiency gains and cost savings from heterogeneous hardware often outweigh the operational overhead. For simple, low-volume use cases, colocated serving remains simpler and sufficient.

What network speed is required for disaggregated serving? Ideally, you need high-bandwidth, low-latency links such as InfiniBand or RoCE (RDMA over Converged Ethernet). For full KV cache transfers of large prompts, 100Gbps+ is recommended to keep TTFT low. Streaming transfer can tolerate slightly lower bandwidth but requires extremely low latency.

Can I mix prefill and decode on the same node? Yes, but this defeats the primary benefit of disaggregation. You may implement a hybrid approach where some nodes handle both, but pure disaggregation offers the best scaling characteristics.

Call to Action

Audit your current KV cache transfer latency. Review your cluster's network topology for disaggregation readiness and identify if your current colocated bottlenecks are worth the migration cost.