How Mixture-of-Experts Changes Your Serving Math

When you deploy a dense Large Language Model (LLM), your serving math is straightforward: every input token triggers a matrix multiplication with every parameter in the model. If a model has 70 billion parameters, inference consumes resources proportional to that full count. Mixture-of-Experts (MoE) architectures fundamentally break this linear relationship. By routing tokens to specialized sub-networks, MoE models allow you to scale parameter counts into the hundreds of billions while keeping active parameters constant.

For infrastructure engineers, this shift changes the bottleneck. You are no longer just managing FLOPs; you are managing memory bandwidth, network topology, and load balancing. This post breaks down the mechanics of that shift.

The Mechanism: Sparse Activation

In a dense transformer layer, the feed-forward network (FFN) applies weights $W$ to every hidden state $h$. The output is $y = W \times h$. In an MoE layer, the FFN is replaced by a mixture of $N$ expert networks. A gating network (usually a simple linear layer with softmax) determines which experts should process each token.

The key constraint is $k$, the number of experts selected per token. Typically, $k=2$ or $k=4$. If you have 16 experts, only 2 are active for any given token. This means the active parameter count is reduced by a factor of $N/k$.

Consider a 560-billion parameter model with 16 experts and $k=2$. The total parameters are 560B, but the active parameters per token are roughly $560B / (16/2) = 70B$. This is equivalent to a 70B dense model in terms of compute. However, the memory footprint remains 560B because all expert weights must reside in GPU memory or be streamed in efficiently.

This decoupling allows you to serve "larger" models without requiring larger GPUs, provided your infrastructure can handle the weight access patterns.

The Cost: Routing Overhead

The gating mechanism is not free. For every token, the model must compute the routing scores, select the top-$k$ experts, and distribute the token to the corresponding expert shards. This introduces two specific overheads: computational and load-balancing.

Computational Overhead

The gating network is small, but it runs for every token. In a typical implementation, the router computes logits for all experts and applies a softmax. This adds a small, constant latency penalty. More importantly, the routing decision dictates data movement. If experts are sharded across multiple GPUs, the router must coordinate with the network to send the token to the correct device.

Load Imbalance

In theory, if the gating network is perfectly balanced, each expert receives $1/N$ of the traffic. In practice, this rarely happens. Certain topics or token patterns may consistently route to specific experts, causing "hot" experts.

If Expert A receives 20% of the traffic while Experts B–Z receive the rest, Expert A becomes a bottleneck. The GPU hosting Expert A will saturate first, forcing the entire batch to wait. This is a common source of tail latency spikes. To mitigate this, MoE implementations often include auxiliary loss functions during training to encourage balanced usage, but at serving time, you must monitor per-expert load.

The Bottleneck: Memory Bandwidth

In dense models, inference is often compute-bound. The GPU spends most of its time performing matrix multiplications. In MoE models, inference becomes memory-bound. Why? Because the active parameters are sparse, but the weight matrices are large.

When a token is routed to an expert, the GPU must load that expert's weight matrix from HBM (High Bandwidth Memory) into the compute units. If the model is sharded across multiple GPUs, the weights for each expert may be distributed. This requires fetching weights from different devices, increasing network traffic.

Consider a scenario where you have 8 GPUs. A 560B MoE model with 16 experts might shard each expert across 2 GPUs. When a token is routed to Expert 1, the system must fetch weights from both GPUs involved. This increases memory bandwidth requirements significantly compared to a dense model where all weights are on a single device (or sharded contiguously).

To handle this, MoE serving requires:

1. High-bandwidth memory (HBM2e or HBM3).
2. Efficient weight loading pipelines (e.g., using CUDA streams to overlap computation and data transfer).
3. Minimal network latency between GPU nodes if experts are distributed.

Infrastructure Implications

For ML engineers, adopting MoE requires changes in how you design your serving stack.

1. Sharding Strategy

In dense models, you typically use tensor parallelism (sharding the weight matrix across GPUs). In MoE, you can use tensor parallelism within each expert, but you also need expert parallelism (sharding different experts across different GPUs). This adds complexity to the communication graph, introducing distributed systems challenges that do not exist in dense model inference.

2. Batch Processing

MoE models benefit from large batch sizes because the routing overhead is amortized. However, due to load imbalance, you cannot simply stack any batch. You must monitor the distribution of tokens across experts. If one expert is overloaded, you may need to dynamically adjust the batch size or re-route tokens. Effective llm-serving infrastructure must account for these dynamic load variations.

3. Monitoring and Observability

You need granular metrics that track:

• Per-expert activation counts.
• Routing latency (time from token input to expert assignment).
• Memory bandwidth utilization per GPU.
• Tail latency (p99) to detect load imbalance.

Standard LLM serving metrics (tokens/sec, latency) are insufficient. You must instrument your system to see the internal routing decisions.

Conclusion

Mixture-of-Experts is not a silver bullet. It reduces active compute, allowing you to scale model intelligence without scaling hardware proportionally. However, it introduces significant complexity in routing, load balancing, and memory management.

For infrastructure engineers, the key takeaway is that MoE shifts the bottleneck from compute to memory and network. If your infrastructure cannot handle high-bandwidth memory access and low-latency inter-GPU communication, MoE will not perform well. Careful monitoring of expert load and efficient weight loading pipelines are essential for successful deployment.

As MoE architectures become more common, expect serving frameworks to evolve with better support for dynamic expert routing and automated load balancing. Until then, understanding the math behind the routing is critical for optimizing performance.