Chinchilla beat a model four times its size on the same compute

A 70B model wins the controlled run

In 2022 a DeepMind team trained two language models on the same compute budget and let the smaller one win. Chinchilla had 70 billion parameters and saw 1.4 trillion training tokens; Gopher, built a year earlier, had 280 billion parameters and saw 300 billion tokens. On MMLU, a 57-subject multiple-choice exam, Chinchilla scored 67.5% against Gopher's roughly 60%, better by more than seven points, from a model four times smaller.¹

That head-to-head is one controlled comparison at one compute budget, and nobody disputes the measurement. Its limit is scope, not validity: one budget, one model family. What it demonstrates is not that small models are better but that Gopher was built wrong for its budget. Gopher saw about 1.07 tokens for every parameter; Chinchilla saw twenty.¹ The result the paper cares about is the allocation rule behind that number: for a fixed budget of training compute, how should you split it between making the model larger and training it on more data?

Where twenty-to-one comes from

The twenty is the agreement of three estimates the paper ran separately and then stacked. Each asks the same question. As the compute budget C grows, how fast should the optimal parameter count N and token count D grow with it? Each answers with an exponent: N_opt ∝ C^a and D_opt ∝ C^b.¹ Approach one reads the minimum off each training curve, held-out loss plotted against compute. Approach two builds isoFLOP profiles, an equal-compute slice in which model size is varied and the loss minimum is read off. Approach three fits a parametric surface, a formula L(N,D) with a handful of constants, to every run and minimizes it.

All three land near a = b = 0.5, and one-half on each exponent has a plain meaning: the two should grow together, so every doubling of the budget should roughly double the model and double the data.¹ At Gopher's budget of 5.76×10²³ floating-point operations, the fitted rule prescribes about 67 billion parameters trained on 1.5 trillion tokens, almost exactly the 70-billion-on-1.4-trillion model the team then built and ran.¹

Why Kaplan pointed the other way

Two years earlier, Kaplan and coauthors had read the same kind of curves and reached the opposite prescription: put almost every new floating-point operation into model size. Their exponents were N ∝ C^0.73 and D ∝ C^0.27, model size growing more than twice as fast as data, with the explicit advice to train very large models on a relatively modest amount of data and stop well short of convergence.² Chinchilla, trained the Kaplan way, would have been enormous and starved of data.

The disagreement is not about data. It is about a hyperparameter. Kaplan held a single cosine learning-rate schedule fixed across every run, whatever its length. A cosine schedule decays the step size that the optimizer takes toward near zero over a preset horizon, so a model stopped early was measured in the middle of that decay. Its intermediate loss overstates what an equally sized model would reach under a matching, shorter schedule. Overstate the loss of the short runs and you understate the value of training on less data, which tilts the optimum toward size.¹ The charge is checkable against Kaplan's own tables: the C^0.73 split is exactly the size-over-data conclusion the schedule artifact predicts, and it comes from the same compute-constrained arithmetic both papers use.² Judged by what 2020 could have known, this is a methodology error, not a lucky guess corrected.

The leg that did not reproduce

The clean "three methods agree" story has one seam. In 2024, Besiroglu and coauthors re-extracted the paper's own data and tried to reproduce the third approach, the parametric fit, and could not.³ The reported exponents were a ≈ 0.454 and b ≈ 0.542, carrying confidence intervals about a thousandth wide. Intervals that tight would demand on the order of 600,000 training runs; the authors had between 400 and 500 data points to fit.³ The reported point estimate implied about 70 tokens per parameter, not 20, out of step with the other two approaches and with the model that was actually trained. Besiroglu's refit of the same surface gave a ≈ 0.512 and b ≈ 0.488, with an interval roughly fifty times wider, and landed back near 20:1.³

The correction is narrower than it sounds. It does not move the ratio. The twenty-to-one figure never rested on the parametric surface; it rested on the isoFLOP minima, the training-curve minima, and the Gopher run, and those are untouched. What Besiroglu removes is the paper's claim that all three legs agreed on their own. As published, one leg was propped up by intervals too narrow to believe. Repaired, it leans the same way as the other two, which is the honest version of the convergence the paper advertised.

Compute-optimal was never cost-optimal

The paper's word for its result is "compute-optimal," and the word is exact: optimal for the one quantity it chose to minimize, training compute. The authors say so plainly.¹ The objective counts the operations needed to train the model and stops there. It does not count the cost of running the model afterward, which is where most of the spending goes once a model serves millions of requests. That gap is not a flaw in the paper; it is a different question the paper did not ask.

LLaMA raised it first. Its objective "disregards the inference budget, which becomes critical when serving a language model at scale," and the team found a 7B model still improving past a trillion tokens, about 143 tokens per parameter, seven times Chinchilla's ratio.⁴ Llama 3 pushed an 8B model to 15 trillion tokens, 1,875 tokens per parameter, ninety-four times the ratio, and reported it still improving log-linearly, against a Chinchilla-optimal figure Meta put at roughly 200 billion tokens for that size.⁵ Sardana and coauthors made the logic formal: fold serving cost into the objective and the optimum slides to models smaller and longer-trained than Chinchilla prescribes.⁶

The reputation grew from the weakest part. When nostalgebraist read the paper as an argument that data, not size, was the binding constraint, the extrapolation ran directly on the fitted L(N,D) surface, the one leg a replication would later fail to reproduce.⁷ Downstream the ratio hardened into a plain rule of thumb, quoted as "15 to 25 tokens per parameter" with no mention of the interval it carried or the runs it was read off.⁸ A fitted optimum was promoted to a settled law somewhere between the tables and the citation.

So "Chinchilla-optimal" carries two different claims. Inside the paper's evidence: at a fixed training-compute budget, spend it near twenty tokens per parameter, and a controlled run backs the number. Outside it sits everything about serving cost, about whether 20:1 is a law rather than one surface's minimum over one set of runs, and about model families the paper never trained. The next time the phrase is offered as the reason a model is the size it is, the useful question is which of the two is meant, and whether the person quoting it is paying to train the model or to run it.