The batch normalization paper got its own explanation wrong

A layer that rescales every mini-batch

Batch normalization is a single step you drop between the layers of a neural network. It takes the numbers coming out of one layer for the current mini-batch, the small group of training examples the network processes together, and reshapes them in two moves. First it subtracts that batch's mean and divides by its standard deviation, so the outputs have zero mean and unit variance. Then it multiplies by one learned parameter and adds another, both trained like any other weight in the network. Those two learned parameters give the network a way to scale and shift the normalized numbers back, so it can recover whatever distribution a layer actually needs, including the un-normalized one.¹

At training time the mean and variance come from the current batch. At inference time the network stops using batch statistics and switches to fixed population values, tracked by a running average over training, so that a single input's prediction no longer depends on which other examples happen to share its batch.¹

What did this buy? The 2015 paper tested it on ImageNet, the 2012 image-classification set of 1,000 categories and about 1.28 million labeled training images that AlexNet had made the field's benchmark.² A batch-normalized version of the Inception network reached 72.2% top-1 accuracy in roughly a fourteenth of the training steps the plain network needed to get there: about 2.1 million steps against 31 million.¹

An ensemble of six batch-normalized networks pushed further, to 4.82% top-5 error on the ImageNet test set, which the paper noted was below the 5.1% error a human rater had scored on the same task.¹ That human number is worth pausing on, because it is not a population average. It comes from one trained annotator who labeled 1,500 of the test images and scored 5.1% top-5 error; a second, less-trained annotator scored about 12.0% on 258 images.² The full test set is 100,000 images. So beating the human meant beating one careful person on a small sample of the task.

The moving target the paper set out to fix

Why would rescaling each layer's inputs help training at all? The paper gave a reason and put it in the title. During training, every layer's weights change at once. So from the vantage of some layer partway up the network, the distribution of numbers arriving from the layers below keeps shifting as those lower layers learn. A layer that has just tuned itself to inputs centered in one range finds, a few steps later, that the range has moved. Normalizing each layer's inputs back to a fixed mean and variance on every step, the argument went, stops later layers from chasing that moving target, so they learn faster.¹ This is a cousin of the distribution shift that degrades a deployed model, except it happens inside the network, during training.

Ioffe and Szegedy gave the effect a name and a precise definition.

A shifting input distribution sounds like a problem, and holding it steady sounds like a fix.

Adding the shift back did not slow training

The only trouble with that reason is that it does not survive a direct test. In 2018 a group at MIT built the experiment the story invites. Santurkar, Tsipras, Ilyas, and Madry took a network with batch normalization and, right after each batch normalization layer, added fresh random noise: drawn with a non-zero mean and a variance that changed every step. This deliberately produces exactly the thing the theory says to avoid. The distribution of each layer's inputs now lurches from step to step, more violently than in a network with no batch normalization at all. If the theory were right, that network should train badly. It trained about as fast and as accurately as the clean batch-normalized one.³

Santurkar's team also measured internal covariate shift directly, though in a different sense than the original paper. Ioffe and Szegedy had defined it as a change in the distribution of a layer's inputs. Santurkar redefined it in terms of the optimization itself: how much a layer's own gradient changes between the moment before and the moment after the layers beneath it are updated on the same step. By that gradient measure, batch-normalized networks often carried as much internal covariate shift as plain ones, sometimes more, and still trained better.³ The two definitions are not the same quantity, and it is worth keeping them apart. But the noise experiment strikes the paper's own distributional version head-on: it drives that distribution shift up on purpose, and training is unmoved.

What batch normalization did change, Santurkar argued, was the shape of the optimization landscape, the surface that gradient descent walks down to fit the network. With batch normalization the loss and its gradients varied more smoothly, so a step in the direction the gradient points lands closer to where the optimizer expects. On one network they trained, VGG on CIFAR-10, the gradient predicted the actual change in loss about two orders of magnitude better with batch normalization than without.³

Three teams, three different reasons

Smoothing is a cleaner story, but the same paper undercuts it as a complete answer. Santurkar's group showed that plain normalizations that make no attempt to control the distribution of a layer's inputs, dividing instead by the L1, L2, or L-infinity size of the activations, smooth the landscape as well as batch normalization and sometimes more.³ Smoothing is real, then, but it is not something only batch normalization does, so it cannot be the whole reason. A separate group, Bjorck and colleagues at Cornell, put the credit somewhere else again. What batch normalization mainly buys, they argue, is the freedom to train at higher learning rates without activations and gradients blowing up as the network deepens. That higher rate is what drives the faster convergence and the better generalization.⁴ A widely used deep-learning textbook, surveying the argument from outside any of these camps, records that the real reason batch normalization works remains an open question.⁵

There is even a setting where batch normalization plainly hurts. Because it estimates a batch's mean and variance from the examples in that batch, it needs the batch to be reasonably large. Wu and He showed that on ResNet-50 with a batch size of two, the error climbs steeply as those estimates get noisy: their batch-size-independent alternative scored 10.6 percentage points lower error at that size.⁶ That is a limit on when batch normalization can be used, and it leaves the debate about why it works untouched.

Through all of it the technique itself keeps working exactly as the 2015 paper showed. The networks train faster, the accuracy numbers hold up, and batch normalization remains in constant use a decade on. The part that did not survive is narrower: the one-sentence reason the paper gave for the speed. And that reason has outlasted the experiments against it. "Internal covariate shift" is still the phrase most people reach for to explain why batch normalization works, years after the test that should have retired it.⁵