Every trained model is rolling downhill on a computed slope

The claim inside “trained by gradient descent”

When a paper says a model was trained by gradient descent, it is describing a picture that happens to be literally true. The model starts somewhere on a landscape of error and rolls downhill, one step at a time, toward a low point. The landscape is the objective; the height at each location is how wrong the model currently is; and each step moves in the direction that lowers that number fastest. Cauchy wrote the method down in 1847, as a way to solve systems of equations, and its shape has not changed since: from wherever you stand, read the slope, and step against it.¹

The one thing to hold onto is where the downhill direction comes from. The method does not try a handful of directions and keep whichever looked best. It computes the slope directly, from the derivative of the objective, and the derivative hands back the steepest downhill direction in a single evaluation. The slope is computed, not searched. That is what makes the idea general: any objective you can differentiate, you can improve this way, without ever solving for the minimum in closed form.¹

Rolling a parabola downhill by hand

Take the simplest bowl there is: f(x) = x², a parabola whose lowest point sits at zero. Its derivative is 2x, and that expression is the whole slope, its size telling you how steep the curve is where you stand and its sign telling you which way is uphill. At x = 1 the slope is 2, positive, so uphill runs to the right and downhill runs to the left. Step left.

How far left is the one free choice, the step size, written η. Multiply the slope by η and subtract, and the new position is x minus η times 2x. Start at x = 1 with a step size of a quarter. The slope there is 2, a quarter of that is one half, and the first step lands at 0.5. From 0.5 the slope is 1, so the next step of 0.25 lands at 0.25. The one after that lands at 0.125, then 0.0625. The position runs 1, 0.5, 0.25, 0.125, 0.0625, each step covering exactly half the ground the last one did, the way folding a strip of paper in half again and again leaves you a smaller and smaller piece.

The error falls faster than the position. The loss runs 1, then 0.25, then 0.0625, 0.015625, 0.00390625, dropping to a quarter of itself each step while x only halves. The motion also slows on its own: because the slope shrinks as x nears zero, each step is smaller than the last even though η never changes. The run eases into the minimum instead of slamming into it. This is the part Cauchy established in his note. For a small enough step the function is guaranteed to come down, and the amount it drops is proportional to the squared steepness of the slope.¹

The gradient points the steepest way down

A real model has millions of numbers to adjust, not one, so the single slope 2x becomes a list: one partial derivative per parameter, each measuring how the error changes as you nudge that parameter alone by itself. Stack those slopes into a vector and you have the gradient, written ∇f, which reads in plain words as the list of slopes pointing the steepest way uphill, so its negative points the steepest way down.³ From there the recipe is the toy example again: step against the gradient, with the step size a separate dial you either fix in advance or set by testing a few lengths along that direction.³

Two practical wrinkles sit on top of this without changing the picture. The gradient of a real objective is usually not computed on all the data at once but estimated from a small sample, and the step size is deliberately shrunk as training goes on. That combination predates neural networks: Robbins and Monro built exactly this scheme of shrinking steps and noisy measurements in 1951, to chase the root of a function they could only sample.⁴ None of it moves the mechanism. The field's standard framing still describes training as pushing the parameters opposite the gradient, with the learning rate, its everyday name for the step size, setting how big each push is.⁵

One number decides whether it settles or flies apart

The step size is a real dial because turning it too far breaks the method in a specific way. Go back to x² and change nothing but the step: set η to 1.5. Now the first step overshoots the bottom and lands on the far side, at −2. The next overshoots worse, to 4, then −8, then 16. Each step doubles the distance to the minimum and flips to the opposite side, so the position runs away while swinging across the very point it is meant to reach. The loss tracks the flight: 1, 4, 16, 64, 256, quadrupling every step.

For this parabola the boundary is exact. Any step size above 1 diverges, and anything below 1 converges. Right at the hinge, η = 1, the position bounces between 1 and −1 forever, coming no nearer and going no farther. A step size of exactly one half is the sweet spot that drops onto the minimum in a single move. The dial has a hard edge, and for x² it sits at one.

The number 1 is special to x². For any other objective the safe range depends on how sharply the surface curves, but the general statement has the same shape as the toy: there is an upper bound on the step size, set by the curvature of the objective, and above it the guarantee of descent is gone.⁶ On the jagged error surfaces of real networks this failure has a name. Such surfaces carry near-vertical walls, and one ordinary step taken at the base of a wall can fling the parameters far across the landscape, throwing away much of the training in a single move.⁷

Where the downhill picture stops

Inside a bowl the downhill picture is exactly right, and it is worth leaning on completely while it holds. The fence is where the surface stops being a single bowl. Gradient descent does reach a low point, but low is local: it settles wherever the ground first goes flat around it, which is a stable resting place and not necessarily the deepest one on the whole surface.⁷

On the high-dimensional surfaces of real networks, the more common obstacle is a saddle: a point where the surface falls away in some directions and rises in others, ringed by wide, nearly flat plateaus. Dauphin and colleagues argued that these saddles, far more than local minima, are what dominate such surfaces, and that the flat regions around them can slow learning to a crawl while looking, from the inside, like a stuck minimum.⁸ The honest form of the caution is about stalling. On the flat the steps grow tiny and progress nearly stops, though gradient descent often does drift free of a saddle in time and carry on down.⁷

One more boundary: the error the method drives down is measured on the training data, which is only a stand-in for the performance anyone actually cares about.⁶ So the next time a paper reports a model trained by gradient descent, the phrase has a machine behind it. A point on a landscape of error reads the slope beneath it and steps downhill by an amount someone chose, easing toward a bottom that is stable, reachable, and only ever as good as the landscape it was given.