MoE (Mixture of Experts)

Also: mixture of experts · active parameters

Mixture of Experts is a model architecture where only a subset of parameters — the active "experts" — fire for each token, so a model can carry huge total parameters while activating far fewer per token. Cost and speed track active parameters; capability tracks total, which is why MoE models can be both large and relatively cheap to serve.

← All terms · Definitions maintained by Mintok. Methodology.