Conditional Compute for Large Language Models
Modern large language models (LLMs) are often trained as dense transformers: every forward pass activates all parameters regardless of input complexity. As parameter counts scale into the hundreds of billions, serving and training dense models becomes prohibitively expensive. Mixture-of-Experts (MoE) architectures address this by replacing certain feed-forward layers with routing mechanisms that activate only a small subset of parameters—called experts—for each token.
This conditional compute approach lets a model grow its total parameter count without proportionally increasing per-token compute, improving efficiency for both training and inference.
Architecture: Gating Networks and Expert Dispatch
An MoE layer typically consists of a gating network and multiple expert sub-networks. For each input token:
- Routing: A lightweight gating network computes weights over available experts.
- Selection: Top-k routing selects one or two experts per token (commonly top-1 or top-2).
- Dispatch: Tokens are sent to their selected experts for parallel processing.
- Combine: Expert outputs are weighted by gate scores and summed.
The unused experts consume no compute for that token, meaning the effective parameter count is lower than the total stored weights. This distinction matters for GPU memory, FLOPs, and inference latency.
Load Balancing and Capacity Constraints
Without constraints, a gating network might route most tokens to a single expert, creating hotspots and underutilizing others. Production MoE systems apply load-balancing losses during training to encourage uniform expert usage. They also enforce per-expert capacity caps; tokens exceeding capacity may be dropped and routed to a shared dense expert or recomputed.
These tradeoffs introduce design knobs: more experts increase model capacity but complicate balancing; higher top-k improves accuracy at the cost of compute.
Deployment Implications
MoE models require specialized serving. Because only certain experts are active per token, naive sharding can leave GPUs idle while others are overloaded. Expert parallelism distributes expert parameters across devices, and routing-aware scheduling helps batch tokens destined for the same expert together.
# Conceptual: MoE-aware serving configuration
# --expert-parallel-size 4 --top-k 2 --expert-capacity 32
For edge and mobile deployments, MoE can reduce per-token memory bandwidth, but only if experts are lazily loaded or quantized effectively.
