AlexNet beat the last hand-built vision system by ten points

Classifying a million images into a thousand bins

The task was narrow and hard. The ImageNet challenge handed a model 1.2 million labeled training photographs and asked it to sort each test image into one of 1,000 categories, scored two ways: top-1 error counts an image wrong when the true label is not the model's single best guess, and top-5 error counts it wrong only when the true label is missing from the model's five most probable guesses. Those images were a slice of a larger database. ImageNet, assembled by Deng and colleagues from crowd workers and organized on the WordNet hierarchy,² held over 15 million labeled images across some 22,000 categories;¹ the challenge used a 1,000-category subset.

Before this paper, the strong entries did not learn their features. They ran hand-designed detectors over the image, summarized the responses into a fixed vector, and fed that to a shallow classifier. The paper's answer was to learn the features instead. A convolutional network learns small filters that slide across the image and report where a pattern appears; stacking such layers, each combining the previous layer's outputs, lets later layers assemble edges into parts and parts into objects. "Deep" names the stack: here, five convolutional layers and three fully-connected layers, eight learned layers holding 60 million parameters, trained end to end on the raw pixels.¹

The result is a pair of numbers, not one. On the 2010 test set the network reached 37.5% top-1 and 17.0% top-5 error; a variant won the 2012 contest at 15.3% top-5 against a 26.2% runner-up.¹ The reconstruction that follows asks two questions of that result: which of the network's parts produced it, judged against the paper's own measurements, and which of those parts the field kept.

Two numbers, two contests

Keep the two error lines apart, because the folklore fuses them. The 37.5% / 17.0% pair is the ILSVRC-2010 result, the one the abstract leads with and Table 1 records. The celebrated 15.3% is a different line entirely: the winning top-5 error in the 2012 contest, from an ensemble of seven networks, two of them pre-trained on the extra Fall-2011 images. Without that extra data the same group's 2012 entry scored 16.4%, and that 16.4% is the number the challenge organizers log as the year's winner.¹

This matters for the claim people make on the paper's behalf. "Top-5 error fell from about 26% to about 15% in a year" is real arithmetic, but the 26.2% is the 2012 second-best entry, a system built on SIFT descriptors and Fisher vectors, which is to say the prior era's method standing in the same contest. So the honest statement is stronger and more specific than a vague annual drop: the network beat the best non-neural system in its own competition by roughly ten top-5 points. The organizers' independent ledger frames the longer arc the same way, putting the 2010 winner at 28.2% top-5 and the 2012 SuperVision entry at 16.4%, with the extra-data variant at 15.3%.³

The unit chosen because it trained faster

Start with a single neuron. It computes a weighted sum of its inputs, then passes that sum through a squashing function. The classic choice, tanh, flattens toward its limits: once a unit is driven strongly positive or negative, the function is nearly flat there, so the slope the training procedure needs to adjust the weights is close to zero and learning at that unit crawls. This is saturation. Replace the squash with the simplest possible alternative, pass positive sums through unchanged and clamp negative sums to zero, and the slope is a constant 1 wherever the unit is active, so updates never stall on a saturated unit. That function, f(x) = max(0, x), is the rectified linear unit, ReLU.¹

The evidence is a training-speed figure, and its scope is worth holding onto. Six times faster to a 25% training error is the CIFAR-10 curve; for the full ImageNet network the paper claims the ReLU model trains "several times faster" but shows no matching curve, so the precise multiple is measured off-domain and asserted on-domain. The idea was also not new. Nair and Hinton had introduced rectified linear units two years earlier, but for a different job, generative restricted Boltzmann machines evaluated on the NORB objects and Labeled Faces in the Wild, not large supervised classifiers.⁴ What this paper added was the demonstration that the unit accelerates supervised training at ImageNet scale.

Overfitting, answered twice

Sixty million parameters over 1.2 million images is an invitation to memorize. A model that large can fit the training photographs so closely that it captures their accidents rather than the categories, and then fails on images it has not seen. That failure is overfitting, and the paper answers it twice.

The first answer manufactures variety. At training time the network never sees a full image; it sees a random 224×224 crop of the 256×256 original, half the time mirrored left to right. The exact input almost never repeats, which the paper counts as expanding the training set by a factor of 2,048. A second scheme perturbs the colours, shifting each image's RGB values along the principal directions of colour variation in the data, and the paper measures this alone as cutting top-1 error by over 1%.¹ The 2,048 is a count of distinct crops, not of independent images: the crops of one photograph are heavily correlated, so the expansion buys less than the raw multiple suggests.

The second answer disables the network as it trains. On each training case, every neuron in the first two fully-connected layers is set to zero with probability one half, chosen fresh each time. No neuron can then count on any particular partner being present, so it cannot build a feature that only works in one fixed committee; it must contribute something useful on its own. This is dropout.¹

One caution about standing. Dropout is the paper's reference [10], the authors' own 2012 preprint, where the same group had already shown it cutting a single network on the 2010 subset from 48.6% to a then-record 42.4% error.⁵ That preprint is a primary source for what dropout is and does, but it shares its authors with AlexNet, so it cannot serve as an independent voice confirming the technique. The dropout result rests on one research group measuring its own idea.

A 3GB card forced the split across two GPUs

The split across two graphics cards is often read as an architectural idea. The paper presents it as a hardware constraint. A single GTX 580 carried 3GB of memory, which capped how large a network could be trained on it, so the authors spread the model across two cards, each holding half the convolutional kernels and the two halves exchanging activations only at certain layers.¹

The measured payoff was −1.7% top-1 and −1.2% top-5 against a one-GPU network with half as many kernels per layer. The paper then undercuts its own comparison in a footnote: the one-GPU baseline "is bigger than 'half the size' of the two-GPU net," so, in the authors' words, the comparison "is biased in favor of the one-GPU net." Read the delta as the authors do, then, as evidence that a bigger network helped their system, not as a clean measurement of the split itself. The specific cross-GPU wiring was also awkward enough that the paper notes it made the connectivity pattern "a problem for cross-validation." That is the signature of a workaround fitted to a memory limit, not a principle to carry forward.

The normalization the field later dropped

One more part earned a measured delta and then did not last. Local response normalization sits after the ReLU in certain layers and rescales each unit's activity by a function of the activity of its neighbouring channels at the same position, so a strongly firing feature damps the features near it. The paper frames this as lateral inhibition borrowed from real neurons, and measures it as reducing top-1 and top-5 error by 1.4% and 1.2%.¹ A smaller trick sits beside it: pooling windows that overlap rather than tile, worth a further 0.4% and 0.3%, and, the paper notes, a touch harder to overfit.

Here an independent check is available, and it points the other way. Simonyan and Zisserman, testing deeper networks on the same ImageNet benchmark and sharing no authors with this paper, report that response normalization "does not improve the performance on the ILSVRC dataset, but leads to increased memory consumption and computation time," and drop it from all but one of their models.⁶ Both findings can hold at once, since the effect can depend on the dataset and the surrounding architecture. But AlexNet's number is a single-system delta measured against one baseline, while VGG's is an independent controlled comparison on the same task, and it is VGG's verdict the field followed: later architectures left the normalization out.

The keepers and the dead ends

Four ingredients outlived the contest, and two did not.

That leaves the interpretive question the paper cannot settle about itself: was the win the architecture, or the data and the compute arriving at once? The honest reading credits the convergence. Writing for IEEE Spectrum, the Computer History Museum's Hansen Hsu casts the moment as three ingredients meeting: "In 2012, AlexNet brought together these elements—deep neural networks, big datasets, and GPUs—for the first time, with pathbreaking results. Each of these needed the other." The same piece records Yann LeCun's read of the turn in the field: "Before AlexNet, almost none of the leading computer-vision papers used neural nets. After it, almost all of them would."⁷ The paper agrees against interest, closing on the note that its results "can be improved simply by waiting for faster GPUs and bigger datasets," which concedes that scale, not the architecture alone, was carrying weight.¹

The paper repays a close reading today for a plain reason: it measured its own parts. Those measurements, not the reputation that grew around them, are what let a reader open it today and sort the units the field kept from the ones it quietly set down.