A network's error peaks where it can just fit its training data, then falls again

The sweet spot the textbook drew

Every course on machine learning teaches a warning. A model that is too small cannot capture the pattern in its data and does poorly. A model that is too large learns the training data too well, soaking up its quirks and mistakes, then failing on anything new. That second failure is overfitting: the model has fit its training examples so closely that its accuracy on unseen examples drops. The number that catches it is test error, the fraction of new examples the model gets wrong. Training error is the fraction of the training examples it gets wrong. From here the textbook rule follows. Drive training error to zero and you have almost certainly overfit. Belkin and his coauthors state the corollary a student would recognize: a model with zero training error is expected to generalize poorly.¹ Between too small and too large lies a sweet spot, and the practitioner's job was to find it and stop there.

You have also absorbed a newer slogan, that a bigger model is a better one, from the scaling laws of the last few years. The two ideas sit badly together. In 2019 a team at Harvard and OpenAI put the discomfort in a title: "Deep double descent: where bigger models and more data hurt."² Their claim is that the textbook curve is only the first half of the picture.

A second fall after the peak

Mikhail Belkin, Daniel Hsu, Siyuan Ma and Soumik Mandal drew the missing half and gave it its name. Their double-descent curve keeps the textbook U on the left and extends it to the right. As a model grows it eventually reaches the point where it is just big enough to fit every training example exactly, driving training error to zero. That point is the interpolation threshold. The classical U says error bottoms out somewhere before the threshold and climbs as you approach it. Belkin's curve agrees up to that point, where test error peaks. Then it adds what the textbook left off. Past the threshold, as the model keeps growing, test error falls a second time, often below the best the old sweet spot could reach.¹ They showed the shape on several methods. One was a predictor built from random features of the MNIST handwritten digits, trained on 10,000 examples, where the peak landed exactly as the number of features rose to equal that count.¹

Nakkiran and his coauthors drew the same curve on the networks people actually train. Their headline case is a ResNet18, a standard image classifier, learning CIFAR-10, small color photographs in ten classes. To make the effect stark they added label noise: a set fraction of the training images had their labels swapped for a random wrong class, a cat stamped "truck," fixed for the run. Here that fraction is 15 percent. They then grew the network through a single width setting, k, and recorded its error at each size.

Read the curve from the left. At the smallest widths the network is too weak, and test error is high. It falls to a first minimum near k = 4, the classical sweet spot. Then it climbs again, to a peak near k = 10, exactly where training error is plunging toward zero and the network is becoming just able to fit the noisy training set. Past that peak, as k heads for the standard width of 64, test error drops a second time, all the way to its lowest point on the curve, roughly 0.29. The best network on the graph is the biggest one, and it sits well on the far side of the threshold the textbook told you to avoid.

Why the model at the threshold is the fragile one

Why is test error worst exactly where the model can just fit the data? Nakkiran's team give an argument. At the threshold there is effectively one setting of the network that fits the training set, with no room to spare. Forced to pass through every training point, including the mislabeled ones, it bends itself out of shape to reach them, and that bending is what a test image pays for. Below the threshold the network cannot fit the data at all, so it never chases individual points and picks up only the broad pattern. Above the threshold many different settings fit the training set equally well. The training procedure, ordinary gradient descent, tends to settle on one that reaches the required points smoothly. It folds in the noisy examples without distorting what it predicts elsewhere.² The peak shows up, the paper argues, whenever the true pattern in the data and the model's own assumptions are mismatched.

This is a claim about model size, and it is worth keeping apart from a neighbor. A separate surprise called grokking tracks a fixed network over training time and finds generalization arriving long after the training error settles. Double descent moves along a different axis. The model changes size, each network is trained to completion, and the quantity on the horizontal is how big the model is, not how long it ran.

Where more data made the model worse

If a bigger model wins past the threshold, is more data always a good trade? Not everywhere. The same paper holds the exception. Adding training data pushes the interpolation threshold right: a larger training set takes a larger model to fit exactly. So a network comfortably past the threshold can be dragged back toward the fragile critical region simply by adding data. Nakkiran's team saw this on a Transformer, the standard translation network, learning IWSLT'14 German-to-English. Raising the training set by a factor of 4.5, from 4,000 to 18,000 sentence pairs, raised its test loss, the translation model's measure of error, across a whole band of model sizes.² The title was not for effect. More data hurt, at fixed model sizes, in a measured experiment.

The sharp peak also owes a great deal to the label noise. Nakkiran's team saw every form of the effect most strongly when noise was present. They swept it from 0 to 5, 10, 15 and 20 percent, and the critical region grew from a gentle plateau into a tall peak as the noise rose.² Noise is not strictly required. Clean training sets still produced real peaks for these networks on the harder CIFAR-100 images and for the translation Transformer.² Noise sharpens the peak and moves where it lands, but a clean dataset does not erase it.

How much of this survived

The result is now cited to bury the old fear of overfitting, sometimes as a blanket permit: make the model bigger and generalization takes care of itself. The reframing is genuine. A survey of the field calls the moment a farewell to the bias-variance tradeoff and reports that heavily overparameterized models often beat the best smaller one.³ Popular summaries fold it into the flat line that larger models simply perform better.⁴ Two lines of later work put fences around that strong reading.

The peak is not a law of nature. It is a property of training with little restraint. Nakkiran, Venkat, Kakade and Ma proved that for a simple family of linear models with well-behaved data, an optimally tuned ridge penalty makes test error fall steadily as either the model or the dataset grows, erasing the peak. A ridge penalty is a standard term that discourages large weights. They showed the same tuning eases it for neural nets.⁵ The second qualification is about counting. Curth, Jeffares and van der Schaar argue that in classical methods such as linear regression, trees and boosting, the raw parameter count hides more than one axis of complexity. Measure effective complexity properly, they find, and the apparent second descent folds back into the textbook U.⁶ Their scope is those classical methods. They do not claim the network curves above are an illusion.

None of this was measured on a large language model. The networks here classify small photographs and translate short sentences, on datasets a single lab could run. Reading double descent as the reason today's largest models work extends the curve past anything the papers put on a graph. It also lands in a separate debate, over whether the jumps in large-model benchmarks are real or an artifact of scoring. That debate turns on how ability is measured, a different question from the interpolation peak.