The emergence jump returns when you plot AI skills against loss

The reply that changed the horizontal axis

In March 2024, Zhengxiao Du, Aohan Zeng, Yuxiao Dong, and Jie Tang posted a paper with a flat title, "Understanding Emergent Abilities from the Loss Perspective," later accepted at NeurIPS 2024.¹ It steps into an argument the field has been having for two years about whether large language models gain abilities gradually or in sudden steps.

A 2022 paper led by Jason Wei had named the sudden version. An emergent ability, Wei and his coauthors wrote, is one "not present in smaller models but is present in larger models," so that such abilities "cannot be predicted simply by extrapolating the performance of smaller models."² In 2023 Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo answered that much of the suddenness lived in the scoring.³ Take the same fixed model outputs and grade them with a continuous rule that gives partial credit instead of an all-or-nothing one, and most of the jumps flatten into smooth climbs. Their paper, "Are Emergent Abilities of Large Language Models a Mirage?", won an Outstanding Main Track Paper award at NeurIPS 2023.⁴ This desk walked through that rebuttal in full. Its result, in one line: change the ruler, and the sudden jump becomes a climb you could have seen coming.

Both of those papers put the same thing on the horizontal axis: model size, whether counted in parameters or in the compute a training run burned. Du's paper changes that axis. Plot a task's score against a model's pre-training loss instead, and four demanding tasks stay at chance until loss falls below about 2.2, then climb. The tipping point stays where it is even under the continuous metrics Schaeffer's team used to make jumps disappear.¹

Pre-training loss, the number a model actually minimizes

A large language model is trained on one narrow job: read some text and predict the next token, the next short chunk of a word. Pre-training loss is the model's average error at that job, measured on held-out text it never trained on. A lower loss means a better next-token predictor, and nothing more direct than that. As this desk covered in OpenAI's 2020 scaling work, this number falls along a smooth, predictable curve as a model grows: it slides down gradually, with no steps in it.

Du's first finding is that loss, not size, is the variable that tracks ability. Models of very different sizes that reach the same pre-training loss score about the same on the same downstream task, whatever their parameter count.¹ That is the reason to move it onto the horizontal axis. Size and compute are inputs a lab chooses; loss is the thing those inputs buy, and it is what the model's downstream skill turns out to follow.

The foundation under this is larger than most emergence studies. Du's team pre-trained more than thirty of their own models from scratch, from 300 million to 32 billion parameters, and checked the pattern against outside model families, LLaMA and Pythia, across 12 downstream datasets.¹ The point of owning the models is control over the axis: with their own training checkpoints they can read a model's pre-training loss directly and line it up against what that same model can do.

Below a loss of 2.2, four tasks wake up

Line up all twelve tasks against pre-training loss, which falls to the left as models improve. Eight of them rise from the start: as loss drops, the score climbs, smoothly, with no threshold to speak of. Four behave differently. They sit at chance across the whole high-loss range and only begin to climb once pre-training loss drops below about 2.2.¹ Those four are the hardest of the set: MMLU, a 57-subject multiple-choice knowledge exam; C-Eval, its Chinese-language counterpart; GSM8K, a bank of grade-school math word problems; and GSM8K's Chinese translation.

Keep two things apart here, because a reader new to this will want to collapse them. The horizontal axis moves smoothly. Loss slides down by small amounts as a model gets better at predicting text, and it never jumps. The threshold is in how these four tasks answer that smooth slide: chance, chance, chance, and then a rise, once loss crosses a line near 2.2. On this reading, Du redefines an emergent ability to match: one absent in higher-loss models and present in lower-loss ones.¹

Changing the axis is a different move than changing the ruler

A reader who has just met the mirage argument should be suspicious here. Schaeffer made a jump vanish by changing how the answers were scored. What stops Du from making the jump reappear the same way, by picking whatever measure brings it back? The worry is the right one.

Schaeffer's fix lived on the vertical axis. It held the model outputs fixed and swapped the scoring rule, from all-or-nothing to one that credits a near-miss, and the sudden curve became a ramp.³ Du's move lives on the horizontal axis: keep the task and its scoring, and change what the scores are plotted against. To meet the metric worry head-on, though, Du also re-scored. On MMLU and C-Eval the team replaced plain accuracy with two continuous metrics. The first, CorrectChoiceProb, is the probability the model places on the right option. The second, Brier Score, rewards confident correct answers and penalizes confident wrong ones. Under both, the tipping point near loss 2.2 stayed where it was.¹

Schaeffer showed that a discontinuous metric can manufacture a jump on the size axis. Du shows that on the loss axis, for these tasks, the jump is still there when the metric is continuous, so for this data the threshold cannot be pinned entirely on the ruler.

Why neither paper ends the argument

Two things keep the question open, and the first sits inside Du's own paper. A falling Brier Score is not the same as a task being solved, because Brier also improves when a model merely spreads less probability onto wrong answers. Du are careful about this. A predictor that ignores the question and splits its probability evenly across four options already scores 0.75 on these four-option exams, and their models' Brier stays no better than that 0.75 baseline until loss crosses the threshold.¹ So the continuous-metric result rests on reading the Brier curve against a known random line, not on the curve being smooth and then steep on its own.

The second is that Du and Schaeffer never grade the same thing. Schaeffer's flattening came from tasks in the BIG-Bench collection, run on Google's LaMDA models.³ Du's persistent threshold came from MMLU, C-Eval, and grade-school math, run on their own models. Both use a metric called Brier Score and reach opposite conclusions, because the tasks and the models underneath are different. Both results can be correct at once, which is exactly why one does not cancel the other. Jason Wei, who wrote the original emergence paper, has answered the whole family of metric arguments in a blog post. For a task like arithmetic you want the exact answer and nothing else, so a partial-credit metric is measuring the wrong thing to begin with.⁵ That is a claim about which metric matters for use, and it leaves Du's threshold neither confirmed nor denied.

The mirage side has stayed measured too. Its authors never claimed that emergence cannot happen, only that a metric can manufacture the look of it. Co-author Brando Miranda has put the point plainly in an interview. The size of a jump is a big function of the metric, he said. No one should be surprised by a sharp jump when the scoring credits only an exact-string match.⁶ Read that way, Schaeffer and Du are closer than the headline dispute suggests. One shows a metric can invent a threshold on the size axis. The other shows a real threshold survives on the loss axis. Neither has run the other's tasks to find out where they truly part.