Mixture-of-Experts models increase representational capacity without evaluating every expert for every token. A router selects a small subset of expert feed-forward networks, and their outputs are combined before the next transformer block.
The promise is conditional computation. The cost is systems complexity.
1. Sparse routing
For token representation and experts , a top- router can be written as:
Only selected experts process the token, but all expert parameters still need to be stored, synchronized or available to the serving system.
2. Why load balance matters
If the router sends too many tokens to one expert, that expert becomes a bottleneck while other capacity is idle. If tokens are dropped because an expert reaches its capacity, quality can degrade in ways that are hard to attribute to the language model itself.
Balancing losses, capacity factors and routing noise are engineering controls, not decorative training details. They affect throughput, specialization and reproducibility.
3. Specialization is not automatically expertise
An expert may specialize by language, topic, syntax or a hidden artifact of the data. Inspecting routing patterns can reveal useful structure, but a high routing concentration does not prove that the expert learned a meaningful concept.
Evaluation should compare dense and sparse models under matched active computation, memory, communication and latency budgets.
4. Security and reliability questions
Routing can become a new attack surface. An adversarial input might target a congested expert, exploit a weak expert or create uneven resource use. Logs should expose routing load, dropped tokens, expert availability and changes after adaptation.
MoE is therefore not only a model architecture. It is a distributed service with a scheduler inside the forward pass.
This note is an original systems synthesis of MoE language-model design.
