The biggest models now train with dropout turned down or off

Half the network vanishes on every training pass

Dropout is usually explained as a switch that stops a network from overfitting. What Nitish Srivastava, Geoffrey Hinton, and three colleagues actually described in 2014 is narrower and stranger. During training, each unit in the network is kept with a fixed probability p and otherwise set to zero. The choice is made independently for every unit on every training case, so no single training pass ever runs on the whole network.¹

What is dropped is whole units, not individual weights: a dropped unit is removed for that pass along with all of its incoming and outgoing connections. The dropping is also random and temporary, so a unit that is gone on one case is likely back on the next. The paper keeps hidden units with probability 0.5 and input units with probability 0.8, values chosen on a held-out set.¹

Training on a different random slice of the network each pass would be useless if the finished network could not then be used as a whole. The paper's answer is a scaling rule. At test time the complete network runs once, with every unit present, but each unit's outgoing weights are multiplied by p. A unit kept half the time during training sent, on average, half its signal downstream. Halving its weights at test time keeps that average the same, so the downstream units see the input they were trained on.¹ The paper has been cited more than 44,000 times.²

Training many networks by training one

The idea first appeared two years earlier, in a 2012 report whose title states the reason its authors believed in it: preventing the co-adaptation of feature detectors.³ Their worry was that a unit which can always count on a particular partner being present will learn to lean on it, becoming useful only inside that fixed arrangement. Remove partners at random and a unit can no longer depend on any one of them, so it has to learn something that helps across many different combinations.¹

The 2014 paper gives that intuition a second, sharper form. Choosing which units to keep carves a smaller network out of the full one, and a network with n units contains 2ⁿ such thinned networks. Dropout trains a different one of them on nearly every pass, and because they all draw from the same pool of weights, no thinned network needs its own copy.

The test-time scaling is an approximate way of averaging the 2ⁿ networks, not an exact one. To check it, they also averaged the predictions of many thinned networks sampled at random, and found that the single scaled network matches that Monte-Carlo average once about 50 samples are drawn.¹ So the ensemble story is an intuition backed by an approximation that holds up well in practice, not a theorem about what the network computes.

How far it moved the error rates of 2012

The 2012 report had already claimed records on speech and object recognition.³ The 2014 paper ran dropout across the standard benchmarks of the moment and reported a consistent drop in error on every one of them. These were the datasets the field measured itself against at the time: handwritten digits, house numbers, small object photos, phone recognition in speech, and topic classification in text.

The gains are real and, in vision, large: dropout took the winning ImageNet 2012 network to a top-5 error of 16.4% when the best classical methods sat near 26%.¹ They are also uneven. On Reuters text the error moved only from 31.05% to 29.62%, an improvement the paper calls much smaller than the ones it saw on images and speech.¹ The method was not free either. Dropout raised training time roughly two to three times, the paper reports, because its updates are noisy: each training pass fits a different thinned network, so the gradients do not point at the full one used at test time.¹

The gain shrinks as the data grows

The uneven benchmark gains were not noise, and the paper measured why. The authors trained the same MNIST network on subsets of the data ranging from 100 examples up to 50,000, with dropout and without, and watched the gap between the two. ¹

At the smallest sizes, 100 and 500 examples, dropout gave no improvement at all. A network this large overfits so small a training set even with the noise dropout adds, so there is no gain left for the regularization to win. As the training set grew, dropout's advantage rose to a peak. Then, as the set grew larger still, the advantage narrowed again, because a network with enough data overfits less on its own.¹

Dropout's value is not a fixed property of the technique. It depends on the ratio between how much a network can memorize and how much data it is shown, and the paper measured a regime on each side where the technique buys little. The benchmarks of 2012 sat in the middle of that range, where models were large relative to the data available, which is exactly where dropout pays.

Where the largest models leave it off

The same curve, extended toward far larger datasets, describes how dropout is used now. The 2017 Transformer, the architecture behind today's language models, kept dropout but cut the rate to 0.1 for its base model, reserving 0.3 for one large translation model. Its own ablation still calls dropout very helpful in avoiding overfitting.⁴ The technique carried forward into modern networks, at a fifth of the original rate.

It also ran into a rival. Once batch normalization became standard in vision networks, the two were found to interfere. A 2018 analysis showed a DenseNet on CIFAR-100 falling from 77.42% accuracy to 68.55% when dropout at rate 0.5 was added inside each block. The cause was a mismatch between the variance dropout produces at training time and the variance the normalization layer expects at test time. A small dose of dropout, 0.1 to 0.2, applied only after the normalization layers, restored consistent small gains.⁵

The clearest break comes at the largest scale. PaLM, a 540-billion-parameter language model trained on 780 billion tokens, was trained with no dropout during pretraining at all, using a rate of 0.1 only when it was later finetuned.⁶ A 2025 study tested this directly and found that removing dropout improves single-epoch language-model pretraining across language modeling, syntax, question answering, and inference benchmarks, and notes that LLaMA reports no dropout use.⁷ The reason lines up with the paper's own curve. A model that sees each of a near-endless supply of examples only once barely overfits. That is the far end of the data axis, where dropout's measured benefit had already gone flat.

It is not that dropout was abandoned. It is still set to 0.1 in the Transformer, still used to finetune PaLM, and still helpful in small doses after normalization layers in vision. Its role narrowed to those regimes rather than ending. And the picture of dropout being switched off at scale rests on the few labs that publish a setting: PaLM states it, LLaMA omits it, the 2025 study tests it.⁷ Frontier proprietary models do not disclose theirs, so this reads from a handful of disclosed reports rather than a survey of the field.