D-MoE: Distributed Mixture of Experts for Edge LLMs

The label D-MoE is used for more than one idea in the literature. It can refer to a dynamic or deep mixture of experts, while distributed MoE work places routing and expert computation across devices. The distinction should be stated before making a systems claim.

This note uses D-MoE as a practical shorthand for a distributed mixture-of-experts deployment.

1. What is distributed

Suppose the router is hosted at node $r$ and expert $E_i$ at node $v_i$. A token path now includes network transfer:

$$T(h_t) = T_{r} + \sum_{i\in g_t}\left(T_{\text{link}}(r,v_i)+T_{E_i}\right) + T_{\text{merge}}.$$

Sparse activation reduces expert computation per token, but a poor placement can turn communication into the dominant cost.

2. Edge constraints

A distributed deployment must account for:

• heterogeneous accelerators,
• link latency and intermittent availability,
• expert memory residency,
• privacy of token representations,
• failures during a routed request,
• load imbalance caused by correlated queries.

Routing should include availability and cost, not only model affinity.

3. Distributed does not automatically mean decentralized

An architecture can distribute experts across edge nodes while keeping one central router and one authority. That may be the right engineering choice. It is different from a peer-to-peer or decentralized system where coordination and membership are also distributed.

Making this distinction explicit avoids claiming a privacy or resilience property that the architecture does not provide.

4. Reliability and security controls

Use authenticated routing messages, bounded payloads, per-expert health signals and a fallback path when an expert is unavailable. Keep a record of which experts handled each request and how often a route was changed.

An adversary who can influence routing may create denial of service, extract information from a specialized expert or cause systematic quality drift. These threats belong in the evaluation plan from the start.

5. A measured deployment path

Begin with a centralized control plane and distributed expert workers. Measure latency, utilization and failure recovery. Then test whether decentralizing routing improves the target requirement enough to justify its coordination cost.

D-MoE is a systems problem before it is a slogan. The placement of the router, experts and evidence determines its real properties.

This note is an original synthesis of distributed MoE design.

Further reading

• Wireless Distributed Mixture of Experts for Large Language Models