Mixture of Experts in LLMs: Capacity Through Selective Routing

Mixture-of-Experts models increase representational capacity without evaluating every expert for every token. A router selects a small subset of expert feed-forward networks, and their outputs are combined before the next transformer block.

The promise is conditional computation. The cost is systems complexity.

1. Sparse routing

For token representation $h_t$ and experts $E_1,\ldots,E_m$, a top-$k$ router can be written as:

$$g_t = \operatorname{TopK}(W_r h_t), \qquad y_t = \sum_{i\in g_t} \alpha_{t,i}E_i(h_t).$$If the formula does not fit, focus it and use the left and right arrows, or scroll horizontally.

Only selected experts process the token, but all expert parameters still need to be stored, synchronized or available to the serving system.

2. Why load balance matters

If the router sends too many tokens to one expert, that expert becomes a bottleneck while other capacity is idle. If tokens are dropped because an expert reaches its capacity, quality can degrade in ways that are hard to attribute to the language model itself.

Balancing losses, capacity factors and routing noise are engineering controls, not decorative training details. They affect throughput, specialization and reproducibility.

3. Specialization is not automatically expertise

An expert may specialize by language, topic, syntax or a hidden artifact of the data. Inspecting routing patterns can reveal useful structure, but a high routing concentration does not prove that the expert learned a meaningful concept.

Evaluation should compare dense and sparse models under matched active computation, memory, communication and latency budgets.

4. Security and reliability questions

Routing can become a new attack surface. An adversarial input might target a congested expert, exploit a weak expert or create uneven resource use. Logs should expose routing load, dropped tokens, expert availability and changes after adaptation.

MoE is therefore not only a model architecture. It is a distributed service with a scheduler inside the forward pass.

Total parameters, active parameters and serving memory

An MoE model contains shared and expert parameters. With $m$ experts per layer and $k$ activated, a conceptual approximation is:

$$P_{total}=P_{shared}+mP_{expert},\qquad P_{active}\approx P_{shared}+kP_{expert}.$$If the formula does not fit, focus it and use the left and right arrows, or scroll horizontally.

Latency and memory are not necessarily proportional to active parameters. Serving requires available weights, a KV cache, communication buffers and capacity for concurrent requests. Tokens within a batch can activate different experts, collectively using much of the expert pool.

Switch Transformers is a reference for sparse routing, while Artetxe and colleagues compare MoE and dense models across tasks. Their results do not let us infer arbitrary deployment costs from the architecture name.

Measuring MoE for a bilingual assistant

I propose separating Spanish, English and mixed-language workloads while matching input lengths. Record time to first token, inter-token time, p95 latency, peak memory and per-expert load distribution. Include technical identifiers because tokenization can change both sequence length and routing patterns.

Compare short generation and long context under low and high concurrency. A model efficient in an isolated test can create queues when requests converge on the same experts. If the engine drops tokens because of capacity limits, report this; some implementations use dropless policies with different trade-offs.

A concrete research question

Does quality by language remain stable as workload distribution changes? Routing concentration can provide a diagnostic signal, but it does not establish vulnerability or semantic specialization on its own.

The distributed MoE note adds network costs, and operational LLM metrics place these measurements alongside quality.