Backpropagation's famous paper showed a network building features no one designed

What the four-page letter actually claimed

In the 9 October 1986 issue of Nature, David Rumelhart, Geoffrey Hinton, and Ronald Williams published a four-page letter titled “Learning representations by back-propagating errors.”¹ Rumelhart and Williams were at the University of California, San Diego; Hinton, the corresponding author, was at Carnegie-Mellon. The letter describes what it calls “a new learning procedure, back-propagation, for networks of neurone-like units” that adjusts the connection weights to shrink the gap between the output a network gives and the output it was supposed to give.¹

The step at the center of that procedure appeared already, in gradient descent: training moves each weight a little way down its share of the error. The 1986 letter is not mainly about that step. Its claim is about the layers that sit between a network's inputs and its outputs, the hidden units: units whose values the training sets, but which no one wires by hand to stand for anything in particular. The letter's stated result is that back-propagation makes those hidden units “come to represent important features of the task domain.”¹ The network is handed inputs and correct outputs and nothing else, and the useful features in between are its own.

The hidden units that learned to sort a family

The letter's clearest demonstration of that claim is a puzzle about family trees.¹ The authors built two family trees, one English and one Italian, with the same shape: twenty-four people joined by twelve kinship relations such as father, aunt, and nephew. They wrote each fact as a triple of a person, a relation, and the person it points to. Given Colin and has-aunt, the network was to switch on the units for Colin's two aunts. It saw 100 of the 104 true triples during training and was tested on the four it had never seen.¹

The network solved the puzzle, including the four held-out triples. The interesting part is how. Sitting between the input people and the output people was a layer of six units, and after training, those six units had taken on jobs no one had assigned them. One unit had come to separate the English people from the Italian ones. Another encoded which generation a person belonged to. A third encoded which branch of the family they came from.¹ Nationality, generation, and branch are the features that make kinship questions answerable, and the network had recovered them from nothing but a list of relatives.

The three authors made the same point at greater length in a companion chapter they published that year in the Parallel Distributed Processing volumes, where they named the procedure the generalized delta rule and derived it in full.² There they wrote that they had answered a challenge Marvin Minsky and Seymour Papert had raised in 1969: that networks without trainable middle layers cannot compute even simple logical functions such as exclusive-or.² The hidden units are what met the challenge, and the family-tree units show what they can do.

How small the demonstrations were

For all its later reach, the demonstration was small. The symmetry network the letter shows in full has six input units, two hidden units, and one output.¹ Its task is to report whether a row of six values reads the same backward as forward. Learning it took 1,425 sweeps through all sixty-four possible inputs, with the weights adjusted once per sweep.¹ The trained network is small enough to print every weight on the page.

The family-tree network was larger but still legible: thirty-six input units across five layers, trained for 1,500 sweeps on 100 of the 104 kinship triples.¹ The companion chapter runs the paper's other test problems, the classic toy tasks of the field. Exclusive-or, the function Minsky and Papert had used against earlier networks, was solved with a single hidden unit in 558 sweeps. Four-bit parity took 2,825 pattern presentations. An eight-input network learned to squeeze eight patterns through three hidden units and reproduce them.²

What the papers never give is how long any of this took, or on what hardware.¹² They count sweeps and pattern presentations, never seconds or machines. A network trained today learns from enormous datasets. The whole training set here was sixty-four inputs, or a hundred kinship triples, swept over and over until the weights stopped moving.

The algorithm predates the paper that made it famous

The paper is often remembered as the invention of back-propagation. Read against the record, that is wrong in both directions. The core operation, computing how every input affects an output by working backward through the steps of a calculation, was in print well before 1986. In 1976, Seppo Linnainmaa published a method for exactly this, tracking how rounding errors accumulate through an arbitrary numerical computation; his first-order coefficient, he wrote, “is the partial derivative” of one value with respect to another.³ He was analyzing rounding error and never mentioned neural networks, but the procedure is the backward pass. Two years earlier, Paul Werbos's 1974 Harvard thesis had set out “dynamic feedback,” which he called “a technique for calculating derivatives inexpensively, for use with the classic method of steepest descent.”⁴ Werbos framed it for fitting statistical models and named brains only as an analogy.

So did the 1986 authors overwrite the people who came before them? Read the two documents and the answer is no. In the Nature letter they write that variants of the procedure “have been discovered independently by David Parker” and by Yann Le Cun, both contemporaries.¹ In the chapter they credit the underlying delta rule to Widrow and Hoff's 1960 work and again name Parker and Le Cun.² What they did not do was cite Werbos or Linnainmaa. The plainest reading of that gap is not theft. Linnainmaa and Werbos had published in numerical analysis and in the behavioral sciences, and their work was not yet known in the connectionist corner of the field the 1986 authors were writing for. The miscrediting that later grew around the paper lives in popular memory, not in the paper's own text.

The fuller chronology reaches back further still, through control-theory work in the early 1960s, as the neural-network researcher Jürgen Schmidhuber has documented.⁵ Schmidhuber is himself a party to these priority disputes and argues a strong case for the predecessors, so read him as an advocate. On the split that matters he is careful: he credits the 1986 paper not with the algorithm but with the experimental demonstration that back-propagation yields useful internal representations.⁵ Linnainmaa owns the backward derivative, Werbos owns its use in steepest descent, and Rumelhart, Hinton, and Williams own the demonstration that, run on a network, it builds the kind of features the family-tree units showed.

How the paper's stated limits held up

The paper did not oversell itself. Rumelhart, Hinton, and Williams named its limits plainly. The most serious, they wrote, is that “the error-surface may contain local minima so that gradient descent is not guaranteed to find a global minimum.”¹ A local minimum is a low point in the error that is not the lowest; a network that settles into one stops improving while a better set of weights exists somewhere else. The same sentence reports what they actually saw: “experience with many tasks shows that the network very rarely gets stuck in poor local minima.”¹ The chapter puts it more strongly, calling the local-minimum problem “irrelevant in a wide variety of learning tasks.”² They were as blunt about biology: the procedure, they wrote, “is not a plausible model of learning in brains.”¹

Did later work prove them wrong about local minima? No, and this is where the legend most often misreads the record. In 2014, Yann Dauphin and five co-authors examined the error surfaces of neural networks in high dimensions and found that the thing which slows training is not poor local minima but saddle points: places where the surface falls away in some directions and rises in others.⁶ The genuine local minima, they showed, are rare, and the ones that exist sit at an error very close to the best possible.⁶ A critical point with high error is almost always a saddle, because in high dimensions “the chance that all the directions around a critical point lead upward is exponentially small.”⁶

That result corrects the paper's explanation while confirming its observation. Rumelhart, Hinton, and Williams had reported, from experiment, that their networks rarely got stuck. Dauphin and colleagues supplied the reason the 1986 authors could not: in a network with many weights, being trapped at a genuinely bad minimum requires every direction to point uphill at once, and that almost never happens. Their observation held. The explanation for it arrived twenty-eight years later.