The knowledge distillation paper trained a network to recognize a digit it never saw

Where the word comes from

"Knowledge distillation" is one of the most-used phrases in AI, and it comes from a single paper. In 2015 Geoffrey Hinton, Oriol Vinyals, and Jeff Dean posted "Distilling the Knowledge in a Neural Network," a short paper grown from a workshop talk.¹ Its idea is specific: take a large, accurate, slow model and train a small, fast one to copy not only its answers but the full set of probabilities behind them.

A decade on, the word is everywhere, and much of what it labels is not what the paper did. When DeepSeek released a family of smaller models in January 2025, it called them "distilled" from its large reasoning model.² The method behind that word had by then drifted a long way from the one the paper introduced.

Soft targets

Training a small network to imitate a big one was not new in 2015. In 2006 Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil showed you could compress a whole ensemble of models into one small network by using the ensemble to label a large pile of data, then training the network on those labels.³ The small network matched the ensemble's predictions while running about a thousand times smaller and faster. Hinton's paper credits this work and says it does the compression a different way.¹

The difference is what the student copies. Bucila's small network copied the ensemble's chosen label. Hinton's copies the whole probability distribution behind that label. A trained classifier does not really output an answer. It ends in a softmax, the function that turns its raw scores into a probability for every possible class (covered in an earlier lesson), and normally only the largest of those probabilities is read off as the answer. The paper's claim is that the rest of the distribution, the small probabilities the model assigns to the answers it rejected, is worth keeping.

Those two models get names. The large one is the teacher, the small one the student, and the teacher's full probability distribution, its confidence spread across every class rather than its top pick alone, is the soft target the student trains to reproduce. The ordinary training signal, the single correct label written as a 1 with 0s everywhere else, is the hard target by contrast.

One thing gets in the way: the informative probabilities are tiny. A confident teacher might give the right class 0.99 and split the last 0.01 across thousands of wrong ones, so the very numbers that carry the knowledge round almost to zero. The paper's fix is temperature, the same dial that softens a model's output when it samples text: divide the scores by a number, T, before the softmax, and a larger T spreads the distribution out. The student trains on the teacher's outputs at a high temperature, so the small probabilities grow large enough to learn from, and then runs at the normal setting once it is trained.¹

The threes it never saw

MNIST is the standard first test in this field: 70,000 small images of handwritten digits, 0 through 9, with 10,000 held back to test on. It is a deliberately easy problem, which is what makes it good for showing an effect cleanly rather than for impressing anyone.

Hinton's team trained a large network on it with every trick for squeezing out errors, and it missed 67 of the 10,000 test digits. A small network trained the ordinary way missed 146. Then they trained that same small network on the large network's soft targets instead of the raw labels, and its errors fell to 74, close to the large network's score and about half the small network's original mistakes.¹

The sharper demonstration came next. They deleted every example of one digit, the 3, from the data the student trained on. The student network was never shown a single labeled 3. Then they tested it on all 1,010 threes in the test set. It got 877 of them right, 86.8 percent, for a digit it had never been taught. It had learned the 3 only from the small amounts of "3" probability the teacher placed on images of the other nine digits.¹

The threes still cost more errors than they should have, and the reason was mundane. The student had never seen how often a 3 comes up, so its internal threshold for guessing 3 sat too low. Raising that one number by a fixed amount lifted the score to 996 of 1,010, or 98.6 percent.¹ The headline figure needed that manual nudge. The result that needed nothing was the plain one: a network reading most of a digit it was never shown, out of the teacher's hesitations about the digits it was.

The largest experiment never distilled anything

Two experiments took the method off the toy problem. The first was speech recognition, on the system then serving Google's voice products: a network trained on about 2,000 hours of audio to label 14,000 sound states. Training ten copies of it and averaging their outputs, an ensemble, raised the share of audio frames it labeled correctly from 58.9 to 61.1 percent. A single network distilled from that ten-model ensemble reached 60.8 percent, recovering almost all of the ensemble's gain from one network instead of ten. It also matched the ensemble's word error rate of 10.7 percent, against the baseline's 10.9.¹ That is the method working end to end at real scale: ten models' worth of knowledge folded into one.

The paper's largest experiment did not go that far. On an internal Google dataset of 100 million labeled images across 15,000 categories, a single baseline network took around six months to train. Rather than distill, the team added 61 "specialist" networks, each trained in a few days to tell apart categories the baseline confused, and the combined system's accuracy rose from 25.0 to 26.1 percent.¹ The low numbers reflect the 15,000-way choice. But this is an ensemble result, not a distilled one. The paper states plainly that it had not yet distilled the specialists' knowledge back into a single network.¹ The experiment that best showed the method's motivation, too many categories for one model to hold, is the one where the distillation step was never run.

Distillation became a family of methods

The word did not stay put. Within a year, Yoon Kim and Alexander Rush used "knowledge distillation" for a method that trained the student on the teacher's actual generated sentences, treated as ordinary correct answers, with no softened probabilities anywhere in it.⁴ By 2021 a survey of the field could sort distillation into three different kinds of knowledge a student might copy, with Hinton's soft targets filling only one of the three.⁵ The definition in wide circulation today, the one a newcomer meets first, is just the general idea: transfer what a large teacher model knows to a smaller student.⁶

Two current systems mark how wide the word has stretched.

DistilBERT is Hinton's method, moved to a new kind of model.⁷ DeepSeek's distilled models never touch a probability distribution. The small model reads the big one's finished answers and learns to write the same kind of thing.² Both are called distillation, and by the broad meaning the word now carries, both are. Only one of them does what the 2015 paper described.