How models are built
MoE
Mixture of Experts — an architecture that activates only a slice of the model's parameters per token, so a huge model can run at the cost of a much smaller one (e.g. DeepSeek, GLM, MiniMax).
How models are built
Mixture of Experts — an architecture that activates only a slice of the model's parameters per token, so a huge model can run at the cost of a much smaller one (e.g. DeepSeek, GLM, MiniMax).