Adam's convergence proof was wrong, and the field kept using Adam

Every weight gets its own step size

Plain gradient descent subtracts a fixed multiple of the gradient from every weight: one step size, applied to all of them at once. Adam gives each weight its own step size, and tunes that step size from the weight's own history of gradients. Diederik Kingma and Jimmy Ba's 2015 paper builds it from two running averages.¹ The first is an average of the gradient itself, m_t = \beta_1\, m_{t-1} + (1-\beta_1)\, g_t, which smooths the noisy per-batch gradient g_t into a steadier direction. The second is an average of the squared gradient, v_t = \beta_2\, v_{t-1} + (1-\beta_2)\, g_t^2, a per-weight record of how large that weight's gradients have recently been. Both start at zero, so both are biased toward zero for the first steps; the paper corrects that bias by dividing each by how much of the average has accumulated, giving \hat{m}_t and \hat{v}_t.

The update divides the smoothed gradient by the square root of the smoothed squared gradient, weight by weight.

A weight whose gradients have been large gets a small step, because its \sqrt{\hat{v}_t} is large; a weight whose gradients have been small gets a large step. That per-weight rescaling is what Adam adds. The recommended settings that ship with it are \alpha = 0.001, \beta_1 = 0.9, \beta_2 = 0.999, and \epsilon = 10^{-8}.¹ The paper backed the method two ways. It reported that Adam trained faster than the competing optimizers on its experiments, and it proved Theorem 4.1, a bound on the regret. Regret is the running total of how much worse the algorithm's choices were than the single best fixed choice made in hindsight. The theorem bounds it by a quantity that grows like \sqrt{T} over T steps. The average regret then falls toward zero as training runs. A proof of that kind is what lets a method claim it converges rather than merely works. Adam's paper has since been cited past 170,000 times on one index, a scale almost no scientific paper of any field reaches.²

The models it called large

The experiments behind the empirical claim were small. Section 6 of the paper opens with the sentence “Using large models and datasets, we demonstrate Adam can efficiently solve practical deep learning problems,” and the word dates the paper: these were the models that counted as large in 2015.¹ There are four of them, and the largest network in the paper is a multilayer perceptron with two hidden layers of a thousand units each.

Every task ran on MNIST or CIFAR-10, datasets of tens of thousands of small images, in minibatches of 128.¹ There is no language model here, and nothing within several orders of magnitude of the models Adam trains today. The method now used to fit networks with hundreds of billions of parameters was demonstrated on a handful of problems a laptop can run. Its lasting reputation rests on how well it held up as people scaled far past anything it was tested on.

The counterexample that broke the proof

In 2018 Sashank Reddi, Satyen Kale, and Sanjiv Kumar built a problem on which Adam converges to the wrong answer, and traced the failure to a specific mistake in Theorem 4.1's proof.³ The problem is as plain as an optimization problem gets. There is one number to choose, x, held between -1 and 1. Every third round the loss is C x for some constant C > 2, and on the other two rounds it is -x. Over any three rounds the gradients are +C once and -1 twice. They sum to C - 2 > 0, so the honest direction pushes x down toward -1, the optimum.

Adam does the opposite. Run it here with \beta_1 = 0 and \beta_2 = 1/(1 + C^2). The iterate walks up to x = +1, the single worst point in the domain, and its average regret never falls to zero.³ The reason is the per-weight rescaling. The one large gradient +C is divided by a large \sqrt{\hat{v}}, and the two small gradients -1 by a small one. The arithmetic leaves the rare up-step larger than the two down-steps that should cancel it. The correct total says go down. Adam goes up.

The flaw in the original proof sits in one quantity. Write \Gamma_t for the change in the inverse step size from one round to the next, \Gamma_t = \sqrt{V_t}/\alpha_t - \sqrt{V_{t-1}}/\alpha_{t-1}. For gradient descent and for AdaGrad this quantity is never negative, because their step sizes only shrink, and the 2015 proof assumed the same held for Adam. It does not. Adam's \hat{v} can fall when a large gradient ages out of the running average, which lets the inverse step size rise and \Gamma_t go negative. In the authors' words, “the proof in the original paper of ADAM erroneously assumes that \Gamma_t is positive semi-definite and is hence, incorrect.”³ Their fix, called AMSGrad, changes one line. It normalizes by the largest \hat{v} seen so far, \hat{v}_t = \max(\hat{v}_{t-1}, v_t). That forces the step size to only shrink and restores a valid \sqrt{T} regret bound.³

Why breaking it changed almost nothing

A disproven convergence proof and a published fix should have moved the field to the fix. It did not. AMSGrad is available in PyTorch as an argument on the Adam optimizer, and its default is off: the constructor ships the exact 2015 settings and treats the correction as an opt-in flag most users never set.⁴ Robin Schmidt, Frank Schneider, and Philipp Hennig then ran the largest standardized optimizer comparison to date: fifteen methods across eight tasks in more than fifty thousand runs. No single optimizer won everywhere. Plain Adam remained a strong default that the newer methods, AMSGrad among them, failed to consistently beat.⁵ The correct theoretical fix rarely changed the outcome on a real workload, so the field had little reason to switch.

The deeper reason it was safe to shrug is that the 2015 proof being wrong is not the same as Adam failing to converge. The counterexample runs at \beta_2 = 1/(1 + C^2), which for C = 3 is 0.1, against the 0.999 everyone actually uses. Later work closed the gap that opened. Yushun Zhang and colleagues proved in 2022 that Adam does converge to a neighborhood of critical points once \beta_2 is large enough, with a phase transition from divergence to convergence as \beta_2 rises. They also noted the flaw in Reddi's setup: it chooses the problem after the hyperparameters are fixed, whereas practice fixes the problem first and then tunes.⁶ Alexandre Défossez and colleagues gave Adam a separate convergence bound for smooth non-convex objectives with bounded gradients, though they observe that at the literal default parameters the clean bound does not close.⁷ The honest reading holds two facts at once: the specific theorem the 2015 paper led with was wrong, and Adam still converges under the large-\beta_2 settings practice runs.

The variant that did displace plain Adam changed something else entirely. AdamW, from Ilya Loshchilov and Frank Hutter, applies weight decay separately from the adaptive gradient step. For an adaptive method that is not equivalent to the usual penalty. The single change improved Adam's generalization enough to become the standard optimizer for training transformers.⁸ The correction that mattered in practice was about regularization, not about the convergence proof that a best-paper award had already broken.