ELMo set records on six language tasks without replacing the models it improved

One vector for “bank,” whatever the sentence

A model that reads text has to turn each word into numbers first. Before 2018 the standard way came from methods like word2vec: every word gets one vector, learned once and looked up the same way every time. A table like that has no room for a word that means two things. “Bank” is one row whether the sentence is about a river or a savings account, and “play” is one row in a theater and on a field. “Deep contextualized word representations,” the paper that introduced ELMo, begins from that failure. Its claim fits in a sentence: a word’s representation should be a function of the entire sentence it appears in, not a fixed entry pulled from a table.¹

How ELMo builds a word from its whole sentence

ELMo stands for Embeddings from Language Models, and it is built out of a language model: a network trained to predict the next word in a sentence from the words before it. Train one on enough text and, to predict well, it has to hold some representation of what it has read so far. ELMo runs two of these, one reading left to right and one reading right to left, each a two-layer LSTM. That pair is the biLM, the bidirectional language model.¹

Here is the move. Instead of throwing the language model away after training and keeping only its word predictions, ELMo reaches inside and takes the internal states, the vectors the LSTM layers hold as they pass over a sentence. For a given word those states already depend on every other word around it, because the network read the whole sentence to compute them. ELMo’s representation of a word is a weighted sum of the biLM’s layers at that word’s position, and the weights are learned separately for each task it is used on. The paper found the two layers do different work: the higher layer captures a word’s meaning in context, well enough to tell a word’s senses apart on its own, while the lower layer captures syntax.¹

Two separable things arrived together in that description. One is an idea: a word’s representation should be computed by a language model that has read the whole sentence, and that model can be pretrained once on plain text and reused. The other is a method for delivering it: a specific frozen two-layer biLM whose layer states you weight, add up, and attach to a task model as extra input. ELMo shipped both. To use it, the paper says, you freeze the biLM’s weights and concatenate the ELMo vector onto the word representation your existing model already takes in.¹ The task model itself does not change. ELMo is added to what it reads.

The released model was not large by a current measure, and the paper never states a single parameter count. It gives the shape instead: two biLSTM layers of 4,096 units with 512-dimensional projections, over a character-based input layer, trained for ten epochs on a one-billion-word corpus of roughly thirty million sentences.¹

Six benchmarks moved by one added representation

The test of the idea was breadth. The authors took six established tasks, each with its own strong published model, and for each one added ELMo to that model’s input and retrained the small task-specific part on top. Every one improved, and on every one the ELMo version set a new state of the art.¹ The six were not variants of one problem. They were question answering, judging whether one sentence follows from another, labeling who did what to whom, linking pronouns to what they refer to, finding names in text, and rating a review’s sentiment.

The question-answering benchmark, SQuAD, shows the reach. It is more than a hundred thousand questions written by crowdworkers about Wikipedia passages, each answer a span of text to be located in the passage.² On it, adding ELMo produced the largest gain of the six.

The gains were real and they were uneven. On SQuAD, adding ELMo cut the baseline’s remaining error by about a quarter. On SNLI, the task of judging whether one sentence follows from another, it cut the error by about a twentieth, and the ELMo score passed the previous record by a tenth of a point.¹ The single summary, “state of the art on six tasks,” hides that spread. What held across all six was not the size of the win but where it came from: in every case a pretrained representation was added to a model that was otherwise left as it was.

ELMo was neither first nor alone

It is tempting to read ELMo as the moment word meaning in NLP stopped being fixed, and easy to overstate it. Context-dependent word vectors already existed. A year earlier, CoVe had taken the encoder from a machine-translation model and used its output as context vectors added to other models, improving several tasks.³ CoVe’s vectors came from a model trained on translation, which needs sentences paired across two languages; ELMo’s came from a language model trained on plain unlabeled text, of which there is effectively an unlimited supply. ELMo was not alone in 2018 either. ULMFiT, submitted weeks before ELMo’s first version, pretrained a language model on plain text and then fine-tuned the whole thing for a task, reducing error by 18 to 24 percent on most of six text-classification datasets.⁴

So the accurate credit is narrower than the legend. ELMo did not invent context-dependent representations, and it did not by itself start the era of pretraining. What it did was make language-model pretraining work broadly: one pretrained representation, dropped into six different task models, setting the state of the art on all of them at once. That breadth is what got noticed. In July 2018 Sebastian Ruder called this NLP’s “ImageNet moment,” arguing that practitioners would soon download pretrained language models the way vision researchers download pretrained image models. He named three works together as the evidence: ELMo, ULMFiT, and the OpenAI transformer.⁵ The phrase is often pinned on ELMo alone now. Ruder pinned it on all three.

The method BERT replaced in a year

Within the year, ELMo’s delivery method was gone. In late 2018 BERT sorted the new pretraining approaches into two kinds. The feature-based kind, and it named ELMo as the example, keeps a task-specific model and feeds the pretrained representation in as extra features, which is exactly what ELMo did.⁶ The fine-tuning kind adds almost no task-specific parts and instead trains all of the pretrained model’s weights on the task itself. BERT was the second kind, and it beat ELMo across the tasks they shared.

BERT’s paper is direct about which part of ELMo it was setting aside.

The complaint is about the delivery, the frozen biLM and the concatenation. It comes with a claim: a single model trained to read in both directions at once is more powerful than two one-directional models glued together after the fact.⁶ It is not a complaint about the idea. The idea BERT kept and built on: pretrain a model on plain text, then reuse what it learned to represent language for some other task.

ELMo’s idea and ELMo’s method ended up in different places. The idea, that a word’s representation should come from a language model pretrained on unlabeled text, is the ground the models after it stand on. ELMo’s method for shipping that idea, a frozen two-layer biLM whose layer states you weight and concatenate as features, lasted about a year in frontline use before fine-tuning a whole Transformer took its place. The first part the field kept. The second it did not.