Standard MoE
(baseline MoE)
learns syntactic / function-word clusters
Clusters · sorted by token count
EMO
(two-level MoE)
learns topical / semantic clusters
Clusters · sorted by token count
Click a cluster on the left to see documents with that cluster's tokens highlighted.
Standard MoE(baseline MoE)
EMO(two-level MoE)
Standard MoE · clusters in this doc
EMO · clusters in this doc