Google's 2017 layer split a model's size from what it computes

Thousands of experts, most of them switched off

In January 2017, seven researchers at Google published a paper with a deliberately immodest title. It was called "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer."¹ Two of the authors were Geoffrey Hinton and Jeff Dean, and the layer they described has since become the standard way to build a very large language model. The idea behind it is plain. Instead of one part of the network processing every word, you build many parts and let each word use only a few of them.

Each of those parts is an expert: a small block of the network that multiplies the numbers standing for one token through a couple of layers of weights and hands back new numbers. (Those incoming numbers, the token's vector, are taught in an earlier lesson.) A normal network, called dense, runs every one of its weights on every token that passes through. The mixture-of-experts layer breaks that habit. It holds a large pool of experts, up to thousands of them, and a small controller that picks a few experts for each token and leaves the rest untouched.¹

The paper's headline result shows why the split is worth the trouble. On a standard language benchmark, one of their models carried 4.3 billion parameters yet spent only 8.9 million operations on each timestep, meaning each position in the text.¹

The best earlier model on that benchmark scored slightly worse while using 151 million parameters and 151 million operations per timestep.¹ The mixture-of-experts model held roughly 28 times as many parameters and did its work in about 17 times fewer operations per step. Size and cost, which normally climb together, had come apart.

How the gate decides which experts run

The controller that picks the experts is the gate, also called the router. For each token it scores every expert in the pool, keeps the few highest scores, and turns those into weights. The experts it did not keep get a weight of zero, and an expert with a weight of zero is never computed. Because it lights up only a handful of many experts, the layer is sparse: most of its parameters sit idle for any given token, which is exactly why a 4.3-billion-parameter model can run on a few million operations.

The output of the layer is a weighted sum of the experts it chose.

Shazeer's team scored the experts with a scheme they called Noisy Top-K. The gate multiplies the token by a small weight matrix to give each expert a score. It adds a little random noise to those scores, which keeps the choice from getting stuck on the same experts every time. Then it keeps the k highest scores and turns them into weights that sum to one.¹ They set k to 4: four experts run for each token, drawn from a pool that could hold thousands.

A gate like this has a failure mode. Left to itself it collapses onto a few favorite experts, trains only those, and wastes the rest of the pool. The paper heads that off with two extra terms added to the training goal. One measures how unevenly the experts are chosen, the other how unevenly the load lands on them. Both push usage to spread across the pool.¹ Keeping the experts balanced turns out to be the hard part of the design, and it is a problem every later paper here inherits.

What a parameter count is really counting

The 2017 benchmark measured cost in operations per timestep. A modern model usually reports two parameter counts instead. The total count is every parameter the model owns, all of its experts added up. The active count is the parameters a single token actually passes through, its slice of the pool plus the parts of the model that are not experts. In a dense model the two are the same number. In a mixture of experts they are not, and the paper's own results table is where they first pull clearly apart.

The gap only widens with scale. In 2021 the Switch Transformer took the same design to 1.571 trillion parameters. It still ran fewer arithmetic operations per input sequence than a conventional dense model of 11 billion parameters, 890 billion against that model's 6.3 trillion.² Roughly 143 times the parameters, and less of the arithmetic. On paper the two models sit far apart. In what a token costs to run, the giant is the cheaper one.

Shipped models you can download report the split directly. Mixtral, a model Mistral released in 2024, states that each token has access to 47 billion parameters but uses only 13 billion of them at inference.³ Its nickname, "8x7B," reads like eight times seven, which would be 56 billion. The real total is 47 billion. Only the feed-forward blocks are split into separate experts. The rest of the model is shared across all of them and counted once.⁴ DeepSeekMoE, released the same month, owns about 16.4 billion parameters and activates about 2.8 billion per token, routing each one to two always-on shared experts plus six of sixty-four specialized ones.⁵

None of this makes the larger number a lie. Every parameter Mixtral counts is really there, and it is doing something: it is the knowledge the model can reach for, even though any one token reaches for only part of it. The larger number is measured correctly. It just measures the store, not the work, and it is easy to quote as if it measured both.

Then Switch routed each token to one expert

The primaries do not agree on how many experts a token should visit. Shazeer's paper chose four, and argued you needed more than one: routing to a single expert, they reasoned, would leave the gate without usable gradients to learn from.¹ GShard, a 600-billion-parameter translation model Google trained in 2020, sent each token to two.⁶

The Switch Transformer, in 2021, went to one. "We instead use a simplified strategy where we route to only a single expert," its authors wrote, reporting that top-1 routing preserved model quality, cut the routing cost, and performed better than sending tokens to more.² Noam Shazeer was an author on that paper too, which is the tell that this was a field revising its own earlier guess rather than a rival scoring a point.

One thing none of them settled is how to keep the experts balanced while training. Shazeer needed two corrective terms. GShard added a hard cap on how many tokens one expert may accept.⁶ Switch replaced the two terms with a single one and, to stop the larger models from diverging, computed the router in higher numerical precision and shrank the starting size of its weights.² A later fix, a term that penalizes runaway router scores, arrived after that.⁴ Each paper changed the recipe, and each still described training stability as something it was managing rather than a matter it had closed.