---
title: "What is mixture of experts (MoE) in LLMs? | BroadRouter"
description: "An MoE LLM routes each token through a few expert sub-networks, so only part of the model runs. Total vs active parameters, with Gemma 4 and DeepSeek examples."
url: https://broadrouter.com/glossary/mixture-of-experts/
lang: en
updated: 2026-11-14
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# Mixture of experts (MoE)

Updated 2026-11-14

Mixture of experts (MoE) is a neural network design in which a layer holds many expert sub-networks and a learned router sends each token to only a few of them. An MoE language model has a large total parameter count but uses a small active subset for each token.

Also called: MoE, sparse MoE, mixture-of-experts model, sparse expert model, active parameters

A dense model runs every weight for every token. A mixture-of-experts model runs only the few experts its router picks.

## How it works

Shazeer et al. introduced the sparsely-gated MoE layer in 2017: up to thousands of feed-forward sub-networks, with “a trainable gating network” choosing “a sparse combination of these experts” for each input. In a modern transformer, the feed-forward block of some layers becomes a set of experts. For each token the router scores the experts and sends the token to the top few. Some designs add a shared expert that every token uses.

Two model cards show the numbers:

| Model | Total parameters | Active per token | Experts |
| --- | --- | --- | --- |
| Gemma 4 26B A4B (Google) | 25.2B | 3.8B | 8 active of 128, plus 1 shared |
| DeepSeek V4.1 Flash (DeepSeek) | 552B backbone | 8B in prefill, 16B in decode | 6 routed of 384, plus 1 shared |

Google’s card says the 26B A4B model “runs almost as fast as a 4B-parameter model.”

## Why it matters

-   **Speed and price per token** follow the active parameters, since that is the compute each token needs. That is how an MoE model can be large and still cheap to serve.
-   **Memory** follows the total parameters. Every expert has to sit in GPU memory, so an MoE model needs as much space for weights as a dense model of the same total size. [Quantization](https://broadrouter.com/glossary/quantization/) is a common way to fit them.
-   **Quality** tracks neither number cleanly. Compare models on your own task, not on parameter counts.

## On BroadNet Inference

BroadNet serves BroadNet Gemma 4 VL 26B under the model ID `broadnet/gemma-4-26b-a4b` and describes it as based on Google’s Gemma 4. Its model catalogue lists the production build as Gemma 4 26B A4B with AWQ 4-bit weights and multi-token prediction. It costs $0.10 input and $0.22 output per 1M tokens, and it is the model behind the hourly key. See [Gemma 4 VL 26B](https://broadrouter.com/models/gemma-4-vl-26b/).

[DeepSeek V4.1 Flash](https://broadrouter.com/models/deepseek-v4.1-flash/) is BroadNet’s 1,048,576-token-context model and the only model on the flat plans. Its MoE design is documented by DeepSeek, not by BroadNet. For how open models compare as a category, see [open-weight models](https://broadrouter.com/glossary/open-weight-model/) and [open-weight vs closed models](https://broadrouter.com/blog/open-weight-vs-closed-models/).

## Questions

### What does A4B mean in Gemma 4 26B A4B?

Google's model card says the "A" stands for active parameters. The model has 25.2B parameters in total but activates about 3.8B per token.

### Is DeepSeek V4.1 Flash a mixture-of-experts model?

Yes. DeepSeek's model card calls it a multimodal MoE model with 552B backbone parameters, 1 shared and 384 routed experts per MoE layer, and 6 routed experts active per token.

### Does an MoE model need less memory?

No. All experts must be loaded, because any token may route to any of them. MoE saves compute per token, not memory for weights.

## See also

-   [Open-weight model](https://broadrouter.com/glossary/open-weight-model/) (term)
-   [Quantization](https://broadrouter.com/glossary/quantization/) (term)
-   [Tokens per second](https://broadrouter.com/glossary/tokens-per-second/) (term)
-   [KV cache](https://broadrouter.com/glossary/kv-cache/) (term)
-   [GPU-hour](https://broadrouter.com/glossary/gpu-hour/) (term)
-   [Use the OpenAI Python SDK with BroadNet Inference](https://broadrouter.com/guides/openai-python-sdk/) (guide)
-   [Open-weight vs closed AI models for business: cost, control, risk](https://broadrouter.com/blog/open-weight-vs-closed-models/) (article)
-   [Best open model for coding agents: pick by tools, context and cost](https://broadrouter.com/blog/best-open-model-for-coding-agents/) (article)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-mixture-of-experts&utm_content=page-cta) ·

**Sources**

-   [Shazeer et al., Outrageously Large Neural Networks, the Sparsely-Gated Mixture-of-Experts Layer, arXiv 1701.06538](https://arxiv.org/abs/1701.06538)
-   [Google, Gemma 4 26B A4B instruction-tuned model card](https://huggingface.co/google/gemma-4-26B-A4B-it)
-   [DeepSeek, DeepSeek-V4.1-Flash model card](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash)
-   [BroadNet Inference API reference, model IDs](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-mixture-of-experts&utm_content=sources)
-   [BroadNet model catalogue (lab measurements, September 2026)](https://www.broadnet.ai/inference/models)
