Foundation Models

Mixture of Experts (MoE)

A mixture of experts (MoE) is a neural architecture in which a learned router activates only a small subset of specialized expert subnetworks per input, decoupling total parameter count from per-token computation. Sparse MoE layers, popularized in large language models such as Switch Transformer and Mixtral, allow scaling capacity at roughly constant inference cost, at the price of routing instability and load-balancing complexity during training.

Why it matters for physical AI

Robots need large model capacity but face hard onboard latency budgets; sparse expert architectures offer a path to capable policies whose per-step compute stays compatible with real-time control.