Deep Dive into Mixture of Experts: From 1991 to DeepSeek-V3

Every major LLM lab faces a conundrum: scale vs. cost. Making a dense model bigger makes it smarter, but also makes every token more expensive to generate. In a dense model, every parameter activates on every token, and the compute cost of a forward pass scales ~linearly with parameter count.

Mixture of Experts: The Core Idea

Mixture of Experts (MoE) challenges this with its architecture. Instead of one giant feed-forward network per layer, an MoE layer holds many smaller "expert" networks, and a learned gate activates only a handful of them per token per layer. The result is a model that can carry hundreds of billions of parameters in vRAM, while activating only a small fraction of them for each run.

To give an idea of scale, the DeepSeek-V3 model holds 671 billion params, while MoE enables it to use only around 37 billion of them each time.

1991: Where It Started

The idea of "mixture of experts" predates deep learning as we know it. It was first proposed in a 1991 paper by Jacobs, Jordan, Nowlan, and Hinton, Adaptive Mixtures of Local Experts. The core idea: instead of training one large network to solve a hard task, train several smaller "expert" networks and a separate gating network that learns how to weight and combine their outputs, all trained jointly.

The paper's demonstration was a vowel discrimination task. Given the same speech data, a system of competing experts plus a gating network was compared against a single monolithic network. The learning procedure was shown to decompose the hard problem into sub-problems that each expert could handle individually.

There's an important architectural detail here that's easy to miss, and it matters for everything that follows: this original formulation is what we'd now call a dense-MoE. The gate produces a weight for every expert, and every expert computes an output for every input; they're just combined with different weights. That single detail is why the 1991 paper sits as a conceptual ancestor rather than a direct blueprint for modern LLM MoE.

It establishes gating as a learned, differentiable routing function, which is an idea that survives for the next three decades. However, it doesn't buy any compute savings, because all experts still run on every input.

2017: The Sparse-Gating Leap

The paper that actually bridges 1991 and modern LLMs is Shazeer et al.'s 2017 paper from Google Brain, Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. The key change here: sparse gating. Instead of every expert computing on every input, the gate selects only the top-k highest-scoring experts (4/4096 in this scenario), and only those experts actually run. This is the shift that actually saves compute, and it's the mechanism every modern LLM MoE is based on.

A minimal version of the routing logic looks like this:

# Simplified top-k sparse gating
def moe_layer(x, experts, gate_weights, k=4):
    logits = x @ gate_weights          # [num_experts] raw scores
    top_k_vals, top_k_idx = topk(logits, k)
    top_k_weights = softmax(top_k_vals)  # renormalized over just the top-k

    output = 0
    for weight, idx in zip(top_k_weights, top_k_idx):
        output += weight * experts[idx](x)  # only these experts run
    return output

Need for a Load-Balancing Loss

Sparse gating introduces a new problem. If you train the gate network purely on the downstream task loss, it tends to collapse. Here's the failure mode: a small random edge in initialisation means one expert responds marginally better to some input than another. The gate notices, routes slightly more traffic there. That expert now gets more gradient updates than its neighbours, so it improves further, so the gate favours it even more. Left unchecked, this rich-get-richer loop concentrates almost all traffic onto a handful of experts, while the rest stay undertrained and never develop anything useful.

Shazeer et al.'s fix is an auxiliary loss, a term added on top of the main task loss that penalises uneven expert usage across a batch, nudging the gate toward spreading tokens more evenly across the whole expert pool.

# Simplified load-balancing auxiliary loss
def load_balance_loss(gate_probs, expert_assignments, num_experts):
    # fraction of tokens routed to each expert
    frac_routed = count_per_expert(expert_assignments) / len(expert_assignments)
    # average gate probability assigned to each expert
    avg_gate_prob = mean(gate_probs, axis=0)
    return num_experts * sum(frac_routed * avg_gate_prob)

2020-2021: Google's MoE-for-Transformers Era

The 2017 paper proved sparse MoE worked in an LSTM. The next step was bringing it into the Transformer, inside the feed-forward (FFN) sub-layer of each transformer block, rather than replacing the whole layer.

GShard (2020) was the first major distributed-training MoE story at Transformer scale, using top-2 gating. Each token was routed to its top 2 experts, and GShard combined this with techniques for sharding enormous expert pools across many devices.

Switch Transformer (2021) simplified this further, to top-1 gating, just one expert per token, per layer. The authors argued this was not just cheaper but more stable to train than top-2, while still scaling to trillion-parameter total capacity.

2023: Mixtral and the Specialisation Question

Mistral's Mixtral 8x7B (2023) is one of the first widely-used open-weight LLMs built on MoE, with 8 experts per layer, top-2 gating, and routing decisions made independently at every layer for every token.

The Mixtral paper includes its own analysis of what the experts specialise in, and the finding is striking: at this scale, on general language modelling, they found no clean topical or domain specialisation. No expert cleanly owns "math," or "code," or "French." What structure they did find looked more syntactic or positional than semantic, plus some correlation between consecutive tokens routing to the same expert, which reads more like local context redundancy than deliberate task-based division of labor.

What the Experts Actually Learned

The 2017 paper also did something that turns out to matter a lot for the rest of this story: it looked inside the trained model to see what each expert had specialised in. Table 9 of the paper, from a WMT'14 English→French translation model, shows real, identifiable clustering, but not along the lines you'd guess.

• One expert fires heavily on phrasing like "a leading/critical/central role"
• Another clusters around rapid/quick/swift-type intensifiers
• Another around innovation- and research-adjacent nouns

This is real specialisation, but it's at pattern-level, not domain-level. There's no "the translation expert" or "the grammar expert."

Mixtral 8x7B (2023), at LLM scale on general-purpose language modelling, found no such clean structure. That's the tension this section picks up: the specialisation debate isn't new, and it doesn't have a single settled answer.

There are at least three plausible explanations for why expert specialisation is still debatable:

1. Scale changes what specialises. At LSTM-with-thousands-of-experts scale on a single task, fine-grained syntactic clusters are easy to detect. At LLM with hundreds of experts per layer scale on a much broader data distribution, that same structure might exist but be diluted across dramatically more traffic per expert.
2. Task changes what specialises. Translation is a much narrower objective than general next-token prediction across a huge, heterogeneous corpus. Narrower tasks might simply produce cleaner, more legible specialisation.
3. Methodology changes what you find. "Specialisation" isn't a single measurable quantity; how you probe for it (which tokens you test, which layer you look at, what counts as a "pattern") shapes what you're able to see.

DeepSeek-V3 and the Current State of the Art

DeepSeek-V3 represents the current high-water mark for this architecture, and it makes two specific bets that build directly on everything above.

Fine-grained experts + shared experts

Instead of a handful of large experts, DeepSeek-V3 uses many smaller experts per layer, alongside a small number of always-on "shared" experts that every token passes through regardless of what the gate decides.

Each MoE layer holds 256 routed experts plus 1 shared expert, and the gate selects the top 8 routed experts per token, so every token is processed by 9 experts total out of 257 available in that layer. The idea is to let the shared expert absorb general, broadly-useful knowledge, freeing the 256 routed experts to specialise on narrower, more distinctive patterns.

DeepSeek ran empirical patterns to check if it actually does what it's supposed to. Turning off the shared expert causes a large capability drop, consistent with it having absorbed genuinely general-purpose knowledge. Turning off even a small slice of the top routed experts (as little as 1/16 of them) also meaningfully hurts performance; a real, measurable signal that the routed experts are holding specialised, non-redundant information.

That's a useful data point to bring back into the specialisation debate: whatever these experts are specialising in, removing them provably costs the model something, even if we can't cleanly label what that something is.

Auxiliary-loss-free load balancing

Rather than the balancing loss described earlier, DeepSeek-V3 uses dynamically adjusted per-expert bias terms added to the gate's scores - nudging routing toward underused experts on the fly, without the accuracy cost that a traditional balancing loss can impose.

# Simplified DeepSeek-style routing sketch
def deepseek_moe_layer(x, routed_experts, shared_experts, gate_weights, bias, k):
    logits = x @ gate_weights + bias  # bias dynamically adjusted for balance
    top_k_vals, top_k_idx = topk(logits, k)
    top_k_weights = softmax(top_k_vals)

    output = sum(s(x) for s in shared_experts)  # always-on
    for weight, idx in zip(top_k_weights, top_k_idx):
        output += weight * routed_experts[idx](x)  # sparse, routed
    return output

The efficiency numbers and engineering demands

DeepSeek-V3 carries roughly 671 billion total parameters, but activates only around 37 billion per token. That gap is the whole MoE thesis made concrete: enormous total capacity, a fraction of it touched per token - and at 256 experts per layer, "a fraction" means well under 4% of any single layer's routed experts fire for a given token.

The engineering this demands is substantial. A pool this large only works with serious infrastructure underneath it: DeepSeek-V3 uses wide expert parallelism during training, spreading the 256 routed experts across many GPUs so each one handles a large enough batch of tokens to stay efficient, with token routing communicated over high-bandwidth interconnects between nodes. This is a direct, practical instance of the compute-vs-memory tension.

Compute vs. Memory: MoE Solves One Bottleneck and Sharpens the Other

The "active params" compute gain is real, but it doesn't tell the whole story of how fast MoE models actually feel to use. Being compute or memory bound are two different bottlenecks, and MoE is genuinely lopsided in how it treats them.

Where MoE wins: compute-bound work.
For any single token, only a handful of experts run, so the FLOPs cost per token scales with active parameters, not total parameters. This shows up most clearly during pre-fill, where the model processes the input prompt, typically with large batch sizes and math-heavy workloads. Pre-fill is compute-bound by nature, and cutting the required FLOPs is a direct, unambiguous win. It's also the main driver behind why training a 671B-parameter-class model can cost closer to what a much smaller dense model would.

Where MoE struggles: memory-bound work.
The catch shows up on the other side of the ledger:

1. The full model still has to live in memory. Because any token can route to any expert, every expert's weights need to be resident in vRAM/RAM at all times.
2. Decode is naturally memory-bandwidth-bound, and MoE doesn't help. During decode, generating tokens one at a time, typically at small batch sizes, the bottleneck is moving weights from memory to the compute units, not doing arithmetic.
3. Dynamic, irregular routing adds its own overhead. Because each token can route to a different, unpredictable set of experts, fetching the relevant weights is less regular than in a dense model's fixed access pattern, which adds its own memory-traffic cost.

Who Actually Benefits-and Why Adoption Isn't a Clean Story

This split has a real consequence: MoE's advantages and disadvantages land very differently depending on who's running the model.

1. Independent users and local/open-source deployments are usually running at small batch sizes, often a single user, one conversation at a time. That's exactly the decode-heavy, memory-bandwidth-bound regime where MoE's compute savings barely matter. A dense model of comparable active size can be a more practical fit here, since there's no giant idle parameter pool to store.
2. Large inference providers running frontier-scale models (1T+ total parameters) are the setting where MoE's tradeoffs actually pay off. At that scale, requests are batched heavily across many concurrent users, which pushes the workload back toward compute-bound territory and lets the FLOPs savings