A network's gradient is a product of local slopes chained backward

The slope gradient descent takes for granted

Training a model means rolling downhill on a computed slope: read the derivative of the error, step against it, repeat. The error is not a plain function of the weights. It is a function of a layer, which is a function of another layer, which is a function of the weights, stacked as deep as the network goes. The slope the method needs is the derivative of that whole stack, and where it comes from is the chain rule, which Rudin calls probably the most important theorem about derivatives.¹

The rule has one moving part. The derivative of a composed function is the product of the derivatives of its pieces, each measured at the point the piece receives. A slope through many stacked functions is a slope at each function, multiplied together. Rumelhart, Hinton, and Williams built neural-network training on exactly this in 1986. They computed each weight's effect on the error by applying the chain rule and passing an error signal backward through the layers.²

Two slopes multiplied, checked three ways

Take a chain short enough to check by eye. An inner step triples its input and adds one. An outer step squares whatever it receives. Send the number 2 through: the inner step returns 7, the outer step squares it to 49. Now nudge the input just above 2 and watch how far the output moves in response.

Each step has a rate of its own. The inner step moves its output three units for every one unit of input, everywhere, because tripling is all it does to a change. The squaring step is steeper the farther out it sits: at the value 7 it moves fourteen units of output for every one unit it receives, since near a point the square grows at twice that point. So a nudge of one at the input arrives at the middle as a nudge of three, and a nudge of three at the middle arrives at the output as a nudge of three times fourteen. The output moves 42 for every unit the input moves. Multiply the two local rates, 3 and 14, and the whole chain's rate falls out.

Now the notation can name what just happened. Write du/dx for the inner rate, read aloud as how the middle quantity moves when the input moves, which is 3. Write dy/du for the outer rate, how the output moves when the middle moves, which is 14. The chain rule says dy/dx equals dy/du times du/dx, or 14 times 3, which is 42. Two other routes confirm the number. Squaring the inner step outright gives nine times the input squared plus six times the input plus one, whose slope is eighteen times the input plus six, and at 2 that is 42. Differentiating the square directly gives six times the inner expression, and at 2 that is 42 again.¹ The same two-link rate, dz/dx equal to dz/dy times dy/dx, is the scalar case Goodfellow, Bengio, and Courville write down before generalizing it to a network.³

The error walks back to a weight

Stack one more link and the same multiplication becomes training. Let a weight of 1 scale an input of 2 to give a hidden value of 2. Let a second weight of 3 scale that to give an output of 6. Measure the error against a target of 4 as half the squared gap, which is half of 2 squared, or 2. Three local rates run along this chain. The error moves with the output at the rate of the gap itself, 2. The output moves with the hidden value at the rate of the second weight, 3. The hidden value moves with the first weight at the rate of the input, 2.

Start at the error and carry one number back. The error's sensitivity to the output is 2. Multiply by the rate linking output to hidden value, 3, and the error's sensitivity to the hidden value is 6. Multiply by the rate linking hidden value to the first weight, 2, and the error's sensitivity to that weight is 12. One number, 2, walked back through two multiplications to 12. Differentiating the error with respect to the first weight in one shot gives the gap times the second weight times the input, or 2 times 3 times 2, which is 12. The backward walk and the direct slope agree.

The reason to walk backward rather than forward is cost, not the chain rule. Propagating rates forward, one input at a time, is also the chain rule, and it lands on the same slopes. But the error is a single number and the weights are many, so starting from the one error and sweeping back reaches every weight's sensitivity in a single pass, where a forward sweep would repeat the work once per weight. Baydin and colleagues call the backward sweep reverse-mode automatic differentiation and show it returns the full gradient of a scalar output in one pass, against one pass per input for the forward direction, which is why it is the mainstay of the field.⁴

Seen this way the network is a graph of simple operations, and the backward pass is local. Each node takes the sensitivity handed down from above and multiplies it by its own rate to pass the right number to each of its inputs, as the Stanford course notes put it, with no node needing to know anything beyond its neighbors.⁵ The bookkeeping is cheap. Nielsen puts the price of the whole gradient at about one forward pass through the network followed by one backward pass, so every weight's slope costs roughly what evaluating the network once costs.⁶

The product that builds the gradient also breaks it

The backward pass multiplies one local rate after another, and nothing holds those rates near one. Suppose each link contributes a rate a little under one. A factor of 0.9 carried back through forty links leaves about 0.015 of the number that started the walk. Suppose instead each rate sits a little over one. A factor of 1.1 through forty links multiplies the starting number by about 45. The multiplication that assembles the gradient is the same multiplication that can drive it to nothing or send it out of range.

Pascanu, Mikolov, and Bengio write that a product of many real numbers can shrink to zero or explode to infinity, and so the gradient carried across many steps vanishes or blows up, the vanishing case making it impossible to learn a link between distant events.⁷ Hochreiter and Schmidhuber reach the same result from a separate line of analysis: error signals flowing backward tend to blow up or vanish, growing or decaying exponentially with the number of steps, so that past a depth nothing can be learned in acceptable time.⁸ The same product that lets one backward pass reach every weight is why very deep and recurrent networks were once hard to train at all.

The scalar picture is a faithful guide, but it is a shadow of the real object. A layer carries many numbers at once, so each local rate is not one slope but a grid of them, a Jacobian, and the backward pass is a product of these matrices rather than of single numbers. What decides whether the carried signal grows or shrinks is then not a lone rate but the matrix's largest stretching factor, its top singular value. In the recurrent case, where the same weight matrix repeats at every step, the threshold sharpens to the largest eigenvalue of that matrix: under one and the long-range signal decays, over one and it can explode.⁷ The single slope below or above one is what that condition looks like with the grid collapsed to a point. The mechanism does not change.

So the slope gradient descent rolls down is a product of local rates, multiplied back along the composition from the error to each weight. Read forward, that product is ordinary calculus. Read backward once, it is backpropagation, and the reason a network's whole gradient costs about one extra pass. Run through enough links with rates off one, it is the reason depth had to be earned. Every one of those is the same product, taken in a different direction.