PaLM was trained on a fourteenth of the data its size called for

The biggest dense model of 2022

In April 2022, Google Research published PaLM, short for Pathways Language Model. It was a single neural network with 540.35 billion parameters, the adjustable numbers a model sets during training.¹ It was densely activated, meaning the whole network runs on every token it reads. And it was autoregressive: trained to do one thing, predict the next token in a sequence, and nothing else.¹

Google trained it on 780 billion tokens of text, a token being roughly a word or piece of a word, on 6,144 TPU v4 chips, Google's own machine-learning processors, spread across two connected pods. Two smaller versions, at 8.63 and 62.50 billion parameters, were trained on the same 780 billion tokens, which lets the paper watch what changes as the model grows.¹

The paper's engineering headline was efficiency. On those chips PaLM reached a model-FLOPs utilization of 46.2%, a measure of how much of the hardware's arithmetic capacity a training run actually used, against 21.3% for GPT-3 and 32.5% for DeepMind's Gopher.¹ How that number is built and where it can mislead is its own lesson. What matters here is the plain fact of the scale: PaLM was the largest densely activated language model anyone had published. By a scaling result that appeared a week before it, that size was more than its training data could justify.

The token budget Chinchilla says it should have had

The result comes from a DeepMind paper the field calls Chinchilla, posted to arXiv on 29 March 2022, seven days before PaLM.² Hoffmann and coauthors trained more than 400 models and found that, for a fixed compute budget, model size and the number of training tokens should grow together in equal proportion. They put it plainly: "the model size and the number of training tokens should be scaled equally."² The rough guide their numbers produce is about 20 training tokens for every parameter, revising an earlier set of scaling laws that had leaned toward spending compute on size. The finding has its own lesson; here it is a ruler.

Compute-optimal is worth catching as it goes past. It does not mean the best model you could build. It means the best model for a fixed amount of arithmetic: given so many chip-hours, how should you divide them between a bigger model and more data.² Hold PaLM against that ruler. PaLM has 540.35 billion parameters and saw 780 billion tokens, which works out to 1.44 tokens per parameter.¹ Chinchilla's own model reached 20 to 1: 70 billion parameters trained on 1.4 trillion tokens. Chinchilla's Table 3 estimates that a model near PaLM's size, 520 billion parameters, should see 11.0 trillion tokens to be compute-optimal.² PaLM saw 780 billion. That is about one-fourteenth of what the rule prescribes for its size.

Two things keep that shortfall from being a verdict on the team. First, PaLM's authors were not ignoring Chinchilla. They could not have used it. The two papers landed a week apart, and PaLM's training was long finished by the time Chinchilla appeared, so the undertrained reading is one applied afterward, with a ruler that did not exist when the model was built.¹² Second, undertrained is a claim about efficiency, not strength. On downstream tasks PaLM 540B beat the much smaller Chinchilla, because it spent far more compute overall.¹ Chinchilla's point is narrower and sharper: the same compute, spent on a smaller model and more data, would have gone further. Chinchilla's own paper had already said as much about the models of its day, calling them "considerably over-sized." PaLM, built in the same months, was the largest of them.²

A jump on BIG-bench that later work called a mirage

BIG-bench is a community benchmark built to test language models on tasks they were not trained for. PaLM's paper reports scores on 150 of its tasks, and on the 58 of them that earlier models had also been measured on. PaLM 540B, given five worked examples in the prompt, a setup the field calls in-context learning, beat the previous best score on 44 of those 58. On the aggregate it scored above the average person asked to do the same tasks, though the average person still beat PaLM on 35% of the individual tasks.¹

The paper's more striking claim was about how the scores arrived. Across the 150 tasks, going from the 62-billion model to the 540-billion one, 25% improved by more than 10 percentage points and 15% by more than 20, a sharper rise than the step from 8 billion to 62 billion had produced.¹ A related paper that year, by Wei and coauthors, named this pattern an emergent ability and defined it precisely.

The claim is that scale switches some capabilities on suddenly, in a way you could not have predicted from the smaller models. That claim has been contested at its root. In 2023 Schaeffer, Miranda, and Koyejo argued the suddenness is largely a product of how the tasks are scored.⁴ Many of them use exact-match accuracy: a model gets no credit until its answer is exactly right, so one that is steadily getting closer shows nothing on the graph until it crosses the line, then appears to leap. Measured on a metric that gives partial credit, they report, the same models improve smoothly.⁴ Their examples center on the GPT-3 family, but PaLM's jumps rest on the same style of scoring, so the challenge reaches them. The emergence debate is its own lesson. PaLM's graph is one exhibit in it, not a settled finding.

The chain-of-thought score that needed a calculator

PaLM's most-cited single result is on GSM8K, a set of grade-school math word problems. The number people remember is 58%. It comes with an asterisk the paper is open about.¹ That 58% is what PaLM scored when it was allowed to call an external calculator to do the arithmetic. Prompted instead to reason step by step in words, a technique called chain-of-thought, with no calculator, PaLM 540B scored 54%. The prior record it is often said to have beaten was 55%, set the year before by a fine-tuned system from Cobbe and coauthors.¹ Plain chain-of-thought came in just under it. A citation crediting PaLM with beating the GSM8K record through reasoning alone is crediting the calculator for part of the work.

PaLM's successors stopped naming their size

Google announced PaLM as a "breakthrough," claiming breakthrough performance and breakthrough capabilities on reasoning tasks, and reported that it beat prior models on 28 of 29 English NLP tasks.⁵ The paper reports the discontinuous jumps on BIG-bench. The announcement, describing the same model, calls its scaling "log-linear behavior similar to prior models" and never uses the word discontinuous.¹⁵ The two framings sit inside Google's own materials.

PaLM's size, the thing this lesson has measured it by, is also the last size Google published for a flagship model. The PaLM 2 technical report, in 2023, gives no parameter count for any PaLM 2 model. It says data and model size should now be scaled in roughly equal proportion, which is Chinchilla's rule. And it states that its largest model, PaLM 2-L, is significantly smaller than the largest PaLM while using more training compute.⁶ The Gemini report that followed discloses parameter counts only for its two smallest on-device models, at 1.8 and 3.25 billion, and none for its Ultra and Pro models.⁷

So the paper reads, now, in two directions at once. Its scaling numbers were corrected within a week of publication, and Google's own successor adopted the correction. Its emergence result is one contested exhibit rather than a law. What it plainly showed is still large: 540 billion parameters trained coherently across 6,144 chips, at higher hardware efficiency than any comparable run had reported, ahead of prior models on 28 of 29 English tasks.¹ Cited today as proof that size alone switches on new abilities, PaLM is being asked to carry more than its paper claimed.