The Mechanics

One gradient step drops the loss from 7.0 to 0.225

The rule subtracts from each weight a fraction of its share of the error, and that same rule, threaded back through the layers, is what sets every weight in a trained network.

8 min read2026-07-19

Prior Context

A model's weights are the whole of what it has learned, and they arrive from one operation repeated until it stops changing them. Seeing that operation once, on numbers small enough to check by hand, is what separates training as a mechanism from the model it produces and from the product built around that model, a distinction most arguments about these systems get wrong. Its pieces return with consequence. The residual, the signed amount a prediction misses its target, is the signal every gradient is built from. The learning rate, the size of one step, decides whether a loss falls smoothly or not at all, and it reappears wherever training cost and instability come up. The chain rule, which threads a single step's gradient back through stacked layers, is backpropagation, and it is why the finished weights of an attention head can be accounted for rather than only used.

Prior An attention head is a weighted average that computes its own weights: the query, key, and value matrices it took as given are exactly what a run of this step settles.
Prior Calculus on Computational Graphs: Backpropagation: reading a derivative as a rate a quantity moves at, the picture the gradient here puts to work.

The loss three points cost

A single linear unit predicts \hat{y} = w x + b. Start it at w = 1 and b = 0, and give it three points, (1, 2), (2, 4), and (3, 6). Its predictions are 1, 2, and 3, each short of its target by 1, 2, and 3. Squared, summed, and halved, those misses cost a loss of 7.0. The half is a convention: the sum-of-squares error carries a factor of one-half so that differentiating the square leaves a clean sum with no stray factor of two, and every number below uses it.¹

One step of gradient descent exists to lower that 7.0. The rule is to subtract from each weight a fraction of the gradient of the loss with respect to that weight.² The step is four operations worked in order: predict, price the error, measure each weight's share of it, and adjust. The three points were chosen to keep the arithmetic legible, not drawn from any trained model; they happen to lie on the line y = 2x, so the target the step moves toward is w = 2, b = 0.

The forward pass and the loss it prices

The forward pass runs each input through \hat{y} = 1 \cdot x + 0 and subtracts the target to get the residual, the signed amount by which the prediction misses. Dive into Deep Learning sets out exactly this squared loss for a single linear unit, carrying the same one-half, and its presentation is the convention used here.¹

The starting unit (w = 1, b = 0) at each point: its prediction and the residual, the signed miss.¹
x	y	prediction ŷ	residual ŷ − y
1	2	1	−1
2	4	2	−2
3	6	3	−3

Each residual squared, the three summed, the total halved: \tfrac12\,[\,(-1)^2 + (-2)^2 + (-3)^2\,] = \tfrac12\,[\,1 + 4 + 9\,] = \tfrac12 \cdot 14 = 7.0.

Each weight's share of the error

A gradient is a set of rates: for each weight, how fast the loss changes as that weight alone moves. Read that rate off the loss and it is an accounting of blame. Because the loss is L = \tfrac12 \sum_i (w x_i + b - y_i)^2, differentiating with respect to w brings the square's exponent down and cancels the one-half, leaving a sum over the three points.²

\frac{\partial L}{\partial w} = \sum_i \htmlClass{nb-mc1}{(\hat{y}_i - y_i)}\,\htmlClass{nb-mc2}{x_i}

(\hat{y}_i - y_i)the residual: how far point i's prediction misses its target.
x_ithe input: how strongly that point's error is tied to w.

The w-gradient is a plain sum over the three points, each point's residual times its input.²

The partial with respect to b is the same sum without the input factor, \partial L / \partial b = \sum_i (\hat{y}_i - y_i), because b enters every prediction with a coefficient of one. Both are consequences of the chain rule, the rule backpropagation runs.³ Evaluate them as plain sums.

Each point's contribution to the two gradients: its residual times its input for w, its residual alone for b. The sums are the gradients.²
point	residual r	input x	r·x → ∂L/∂w	r → ∂L/∂b
1	−1	1	−1	−1
2	−2	2	−4	−2
3	−3	3	−9	−3
sum			−14	−6

Point three, with the largest input and the largest miss, contributes −9 of the −14 in the w-gradient: it is the point most answerable for the error the step is about to correct. Reading a gradient this way, as credit assigned for the error, is the standard one.⁴⁵ Nudge w up by a hair and the loss falls at a rate of 14; nudge b up by a hair and it falls at a rate of 6.

The update that lowers the loss

The update rule subtracts from each weight the learning rate times its gradient.² Pick a learning rate \eta = 0.05. Then w \leftarrow 1 - 0.05 \cdot (-14) = 1 + 0.70 = 1.7 and b \leftarrow 0 - 0.05 \cdot (-6) = 0 + 0.30 = 0.30. Both weights rose, because both gradients were negative and the rule steps against them. The learning rate is the one free choice, the size of the step, and it is among the most consequential settings in training: too small and the loss crawls down, too large and the step overshoots the valley it was aiming at.⁶

Recompute the predictions with the new weights, \hat{y} = 1.7 x + 0.3, and price the error again.

The same three points after the step (w = 1.7, b = 0.3): predictions closer, residuals smaller.¹
x	y	prediction ŷ	residual ŷ − y
1	2	2.0	0.0
2	4	3.7	−0.3
3	6	5.4	−0.6

The loss is now \tfrac12\,[\,0 + 0.09 + 0.36\,] = \tfrac12 \cdot 0.45 = 0.225. It fell from 7.0 to 0.225 in one step, checked on a calculator.

One number in the update looks wrong and is not. b rose to 0.3, though the three points sit exactly on y = 2x, where the best bias is zero. With w still short of 2 the unit under-predicts every point, and raising b lifts all three predictions toward their targets, so its gradient is negative and the step raises it. The gradient descends the joint loss over both weights at once, not each weight toward its own answer; once a later step brings w near 2, b's gradient reverses and pulls it back down.² The gradient is also a local rate, exact only for an infinitesimal move, so the predicted fall overstates the real one and the rule is a small step, recomputed and repeated, rather than a single leap to the answer.

An old method, sampled

The step just worked is not new. Deep Learning attributes the method, repeatedly moving opposite the gradient to lower a function of several unknowns, to Cauchy in 1847, and names it steepest descent.² The move against the gradient is the same one hundred and seventy-nine years on.

Two things separate that worked step from how a real unit is trained. The first is scale. The example summed the gradient over all three points, which Ruder names batch gradient descent, and a real training set is too large to sum in full for every step, so training samples it, taking the gradient from one example or a small batch at a time.⁷ The second is what sampling does to the signal. Robbins and Monro set out the scheme in 1951: a sequence of steps of shrinking size, taken against a noisy measurement of the target, x_{n+1} - x_n = a_n(\alpha - y_n), which converges even though each step reads the error imperfectly.⁸ A sampled gradient is such a noisy measurement, and that version, stochastic gradient descent, is what trains the unit.

The chain rule settles the weights

One unit has one layer, so its gradient was a single sum. A network stacks layers, and the loss depends on an early weight only through everything the later layers compute from it. Backpropagation is the same step under that condition: it applies the chain rule to carry the gradient backward from the loss to each weight, one layer at a time, reusing the work at each stage.³ Rumelhart, Hinton, and Williams named it in 1986 as repeated weight adjustment that shrinks the difference between a network's output and the desired one.³ Olah frames the same computation as reverse-mode differentiation on a graph, each derivative a product the chain rule assembles as the calculation moves back from the output.⁹ Backpropagation is only the method for computing the gradient; a separate algorithm, gradient descent on that gradient, does the learning.²

That distinction keeps three things apart that are constantly confused. The weights are the model: the numbers \hat{y} = w x + b uses, or the projection matrices an attention head uses. Training is the loop just worked, the one that set those numbers by pushing the loss down step after step. The product is neither the weights nor the loop; it is the interface, the serving system, and the company built around the numbers. The first two of these are drawn straight from how the gradient is computed and then applied.² The third is a synthesis, not a sourced fact.

That loop is the answer the attention piece left open. The weights it took as given, the query, key, and value matrices whose numbers arrived without a source, are what a run of this step settles to: a small correction at a time, until the gradient over the training set is near zero and the weights stop moving. Training is a mechanism, not a magic word. The proof is that its smallest instance runs on paper and the loss falls, from 7.0 to 0.225, in one checked step.