Adam's convergence proof was wrong and it still trains deep learning

Every parameter gets its own step size

Plain gradient descent moves every parameter by the same rule: take the gradient, multiply by one step size shared across the whole model, subtract. That single number sets the pace for millions of parameters, and tuning it is where practitioners spend the most effort. Adam keeps the shape of that step and changes what sets its size. Each parameter earns its own step, scaled by how large and how noisy its gradients have recently been, and the model-wide learning rate becomes a knob that, at its default, mostly does not need touching.¹

The mechanism is two running averages of the gradient and a small correction for how they behave in the first few steps: a direction from one average, a per-parameter scale from the other, and a rescaling so the earliest steps are not dragged toward zero. The paper then proves the method converges and measures it on a handful of models. Two of those claims did not survive contact with later work.

Two averages of the gradient

Write gₜ for the gradient at step t. Adam does not use it raw. It keeps a running average of the gradient and, separately, a running average of the gradient squared, updating both a little on every step.

The first is the first moment, mₜ. Its update is mₜ = β₁·mₜ₋₁ + (1−β₁)·gₜ, with β₁ = 0.9 by default. Read that as a mixture: keep 90% of the old average, fold in 10% of the newest gradient, so one noisy gradient barely moves it. If the gradients hold steady at some value g, the average settles at exactly g, because at a fixed point m = 0.9·m + 0.1·g. So mₜ estimates the mean gradient. This is momentum: the step follows where the gradients have been agreeing to point.¹

The second is the second raw moment, vₜ, built the same way from the squared gradient: vₜ = β₂·vₜ₋₁ + (1−β₂)·gₜ², with β₂ = 0.999. Squaring throws away the sign and keeps the size, so vₜ tracks the typical magnitude of a parameter's gradients. If the gradients hold steady at g, vₜ settles at g², the uncentered variance. That per-parameter size estimate is the idea Adam inherits from AdaGrad, which first scaled each parameter's step by its own accumulated gradient history.² Adam's twist is to make it a decaying average rather than a running total, so the scale reflects recent gradients, not every gradient since step one.

Why the first steps need rescaling

Both averages start at zero, because there is no gradient history before the first step. That start biases them. On step one with β₂ = 0.999, the second-moment update gives v₁ = 0.001·g₁², a value a thousand times smaller than the g₁² it is supposed to estimate. The average is technically correct as a weighted sum, but it sits far below the quantity it stands for, and it stays low for many steps while the history fills in.

The repair is division. Adam divides mₜ by (1 − β₁ᵗ) and vₜ by (1 − β₂ᵗ), where t is the step number, to get the bias-corrected estimates m̂ₜ and v̂ₜ. On step one the second divisor is (1 − 0.999) = 0.001, so dividing v₁ by it multiplies the estimate back up roughly a thousandfold, undoing the shrinkage exactly. As t grows, β₂ᵗ falls toward zero, the divisor approaches one, and the correction quietly switches off.¹ The correction is not cosmetic: without it, an average that starts far too small makes the update's denominator too small and inflates the step, and the paper reports that skipping this correction, as RMSProp does, "leads to very large stepsizes and often divergence."¹

Assembling the step

With both corrected averages in hand, the update is one line. The direction comes from the smoothed gradient; the size comes from dividing by the square root of the smoothed squared gradient; a tiny constant keeps the division safe.

m_t  = β₁·m_{t−1} + (1−β₁)·g_t          first moment
v_t  = β₂·v_{t−1} + (1−β₂)·g_t²         second raw moment
m̂_t = m_t / (1 − β₁ᵗ)                   bias-corrected
v̂_t = v_t / (1 − β₂ᵗ)                   bias-corrected
θ_t  = θ_{t−1} − α · m̂_t / (√v̂_t + ε)  parameter update

defaults:  α = 0.001   β₁ = 0.9   β₂ = 0.999   ε = 1e-8

Read the last line against plain gradient descent. There, the update was θ minus one shared step size times the gradient. Here the gradient is replaced by m̂ₜ, the averaged direction, and the single step size is replaced by α divided by (√v̂ₜ + ε), which is different for every parameter. A parameter with large, noisy gradients builds a large v̂ₜ, so its denominator is large and its step shrinks. The constant ε = 1e-8 sits inside the denominator only to stop a division by zero when v̂ₜ is tiny. The stated defaults, α = 0.001, β₁ = 0.9, β₂ = 0.999, are the settings the paper recommends across the problems it tested.¹

Each parameter moves in the direction its gradients have been agreeing on, by an amount cut down in proportion to how large those gradients have recently been. One consequence the paper notes is that a step's size is approximately bounded by α regardless of the gradient's scale, so α sets a trust region on how far any parameter can move in one update.¹ Those five lines are Adam's signature: two moving averages of the gradient, two (1 − βᵗ) divisors that undo their early bias, and an ε living under the square root.

What section 6 actually tested

A method this widely trusted invites the assumption that its paper proved it trains better models. It did not test that. The experiments ran three kinds of model: convex logistic regression on MNIST digits and a bag-of-words version of the IMDB reviews, a small network of two hidden layers of 1000 ReLU units each on MNIST, and one modest convolutional stack on CIFAR-10.¹ Every curve in the section plots training cost against steps or epochs. None of them measures error on held-out test data, so the paper measured how fast Adam drives down the training objective, never how well the model generalizes.

On dense MNIST logistic regression Adam matched SGD with momentum and beat AdaGrad; on sparse IMDB bag-of-words, where per-parameter scaling suits sparse features, AdaGrad stayed competitive. On the small network, where the paper's own convergence theory does not apply, it reports that Adam "often outperforms other methods." The convolutional case is where the paper argues against its own headline.¹

The evidence supports a narrow claim: Adam reaches low training loss quickly on these problems, and on the one convolutional case it is barely ahead of a tuned baseline and partly loses the adaptivity it was built for. The broad reputation, that Adam produces models that generalize well, is not what section 6 measured.

The proof a convex problem broke

The paper does not rest on experiments alone. Section 4 offers a theorem: in the online convex setting, Adam's average regret shrinks at the rate O(1/√T), a guarantee "comparable to the best known results." Regret measures how much worse Adam's running choices are than the single best fixed choice in hindsight, so vanishing average regret means its decisions approach optimal over time. That is the claim that failed.¹

In 2018, Reddi, Kale and Kumar built a problem where Adam's average regret does not vanish. It is as simple as such a problem gets: one dimension, a variable confined to [−1, 1], and a loss that on every third step is Cx for a constant C larger than 2, and −x otherwise. The rare large gradient C points toward the true minimizer at x = −1; the frequent small gradients point the other way. Adam converges to x = +1, the worst point in the set. The reason is the second moment: the one large gradient is scaled down by its own square before it is applied, so the many small gradients outvote it.³

The counterexample holds β₁ fixed, while Adam's proof used a β₁ that decays over time, so the settings are not identical. Reddi and colleagues address this: the parameter that drives the failure is β₂, not β₁. So the proof's key assumption is false, and explicit convex problems exist where Adam does not converge. Their fix, AMSGrad, forces the second-moment denominator to never decrease by carrying forward the largest v̂ seen so far, restoring the guarantee the original proof only appeared to establish.³

What "weight decay" in Adam was not

The second failure is quieter and lives outside the original paper's explicit claims. Practitioners regularize large models by shrinking weights toward zero each step, a technique called weight decay. For plain SGD it is exactly equivalent to adding an L2 penalty to the loss, so the two are used interchangeably. For Adam that equivalence breaks.

The mechanism is the same denominator that broke the proof. When the penalty is added into the loss as L2, its contribution enters the gradient and is then divided by √v̂ₜ. A parameter with large recent gradients has a large denominator, so its decay is scaled down; the parameters that most need shrinking are shrunk least. Loshchilov and Hutter's fix, AdamW, moves the decay term outside the adaptive division, so every weight is pulled toward zero by the same proportion regardless of its gradient history. They report the decoupled version closes much of Adam's generalization gap, reaching about 15% relative improvement in test error and letting Adam compete with SGD and momentum on image classification.⁴ This is a defect in Adam as it was widely implemented rather than in the theorem the original paper stated, and it is why AdamW, not Adam, is the default that trains most large models today.

Indispensable and wrong at once

The two failures are different kinds of object, and the difference is one of evidence. The convergence theorem was refuted by a specific counterexample: a named quantity in the proof does not have the property the proof assumed, and a one-dimensional convex problem shows Adam converging to the worst available point. That is a broken proof, full stop. The weight-decay coupling is not a false theorem but a design choice that quietly handicapped generalization, invisible on training curves and visible only when another team compared held-out error. A refuted proof announces itself; a regularizer that silently works a little worse does not, and the second kind of failure is the harder one to catch.

What held up is just as concrete. The two-average update with bias correction became the default first thing to reach for across deep learning, and independent surveys record its exact defaults, β₁ = 0.9, β₂ = 0.999, ε = 1e-8, entering the field as common reference.⁵ The robustness of those defaults is real, and bias correction survived both corrections that followed: AMSGrad and AdamW each keep the two moving averages and the (1 − βᵗ) divisors and change only what happens after. The generalization question the paper never measured was later put to it: Wilson and colleagues found that adaptive methods, Adam among them, tend to generalize worse than plain SGD even when they reach lower training loss.⁶ That finding is the outside view the paper's own experiments could not supply, and it is what AdamW partly answered.

The reason to spend an evening with the paper is its own section 6. The experiments plotted training cost and nothing more, conceded the convolutional result was marginal, and reported the second moment collapsing on exactly the model type that would come to dominate the field. That candor is what lets one algorithm be indispensable in practice and wrong in its central theorem at the same time: the paper never claimed the ground its proof could not hold. Reading section 6 for what it actually reports is how those facts stay separately true instead of dissolving into one convenient story.