Whenever you're ready, walk me through how you'd approach this.
Explain Mixture of Experts (MoE) architectures.
Discuss how a router selects experts for each token, how sparse activation reduces computation, and how training addresses expert load imbalance. Include the roles of top-k routing, capacity factors, auxiliary balancing losses, token dropping, and communication costs. Be prepared to sketch or implement a minimal MoE layer and explain when it is preferable to a dense feed-forward network.