Google packed a whole sentence into a single vector

Reading a sentence into one vector

In 2014, three Google researchers reported a translation system built on a single idea. One neural network reads an English sentence and squeezes it into a fixed-length list of numbers, and a second reads that list back out as French.¹ The authors were Ilya Sutskever, Oriol Vinyals, and Quoc Le. They named the arrangement sequence to sequence, and the name now covers a large share of what neural networks do with language.

Start with the list of numbers, because it is the whole trick. A nutrition label reduces any dish, an apple or a lasagna, to the same short row of numbers, and two foods whose rows are close are alike in the ways the label measures. The encoder network does this to a sentence. It reads the words one at a time, keeps a running summary, and when the sentence ends it hands over that summary: a fixed-length vector, a list of the same length whether the sentence ran three words or thirty.¹ The difference from a nutrition label is size. A label has a handful of fields; the vector here is built inside network layers that each carry a thousand numbers, room enough, the authors bet, to hold everything the second network needs to rewrite the sentence in another language.¹

Both networks are long short-term memory networks, or LSTMs, a kind of recurrent network built to carry information across long stretches of text. The second one runs the first in reverse. Given the vector, it writes French one word at a time, each word chosen from the vector and the words already written.¹ The single most likely word at each step does not always add up to the best sentence, so the decoder keeps a short list of the most promising partial translations and extends them together, a procedure called beam search.¹ Read a sequence in with one network, write a sequence out with another: that is the shape the paper is remembered for.

What 34.81 is actually made of

The system was tested on one task, translating English news text into French, on the standard WMT'14 benchmark.¹ The thing it had to beat was not another neural network. It was a phrase-based statistical translator, the mainstream method of the day, which scored 33.30 on the benchmark's BLEU scale, a 0-to-100 measure of how closely a machine translation matches human reference translations.¹

A single one of the paper's LSTMs, with the source reversed, scored 30.59, below the statistical baseline.¹ The headline 34.81 came from stacking: five separately trained networks voting together, decoded with a beam of twelve.¹ And the best number in the paper, 36.5, was not the LSTM translating on its own. It was that same five-network ensemble used to re-rank the statistical system's own thousand best guesses.¹

The whole model held about 384 million adjustable numbers, most of them not in the translating layers but in two lookup tables: one row for each of the 160,000 English words it could read and each of the 80,000 French words it could write.¹ It trained on 12 million sentence pairs, 304 million English words against 348 million French, on eight GPUs for about ten days.¹ For 2014 that was a large machine, and the entire return on it was one direction of one language pair.

Why they reversed the English

The single change that bought the most was strange. Before training, the authors reversed the word order of every English source sentence, feeding "sat cat the" in place of "the cat sat" while leaving the French target in normal order.¹ On its own that flip raised the single-model score from 25.9 to 30.6 BLEU and dropped the network's perplexity, a measure of how surprised it is by the next word, from 5.8 to 4.7.¹

Their explanation was partial, and they said so. Reversing the source puts the first English words next to the first French words they line up with, shortening the distance the network has to carry an early dependency before it starts producing output.¹ Why one flip helped so evenly across a whole sentence, and not only its opening, they could not fully say. In their own words, "we do not have a complete explanation to this phenomenon."¹

The trick worked and its numbers were real, and a hand-adjustment of that size tells you something. A network that needed its input flipped, to keep early words close to their translations, was feeling the strain of carrying a long sentence through one vector. That is a narrower admission than the one the field later pinned on the paper, and the authors made only the narrow one. On the larger question, whether a single fixed vector was a ceiling, they claimed the opposite.

The bottleneck the paper never conceded

Squeezing a whole sentence into one fixed vector sounds like it should break on long sentences, where there is more to hold. The authors expected that, and found the reverse.¹ Their system's score held roughly flat as sentences grew, out to about thirty-five words, with only slight slippage on the longest.¹

The diagnosis the field remembers as this paper's flaw is not in this paper. It was made the same month, by a different group. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio conjectured "that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder-decoder architecture."² They offered a fix. Instead of one vector for the whole sentence, encode the source as a sequence of vectors, and let the decoder pick which of them to read for each word it writes. That fix is attention.

A companion paper then measured the failure the conjecture predicted. Cho and colleagues tested a plain recurrent encoder-decoder and watched its quality "suffer significantly as the length of the sentences increases."³ They blamed "the fixed-length vector representation," which "does not have enough capacity to encode a long sentence."³ The breakdown was not measured on Sutskever's deep, reversed, five-network system. It was measured on a single-layer model without the reversing trick.³ The bottleneck was a property of the approach, and the particular machine in the seq2seq paper had been tuned, by reversing and depth and ensembling, to hold it off.

The account that reached the textbooks kept the bottleneck and dropped the surprise. A standard deep-learning text now introduces the encoder as turning "a variable-length sequence as input" into "a fixed-shape hidden state," then moves straight to attention as the way past that single fixed-length limit.⁴ Attention won. Within three years the Transformer kept attention and dropped recurrence entirely, "dispensing with recurrence and convolutions,"⁵ and it became the design nearly every large language model now uses.

So what did 2014 actually establish? The durable part was the shape, not the score. Today "sequence to sequence" names a general recipe: read a sequence in, write a sequence out. It covers translation, summarization, and question answering alike. One widely used framework, T5, casts every text task into a single text-in, text-out form.⁶ The founding result under that sweeping name did one direction of one language pair and leaned on a hack a better idea made obsolete. The framing lasted. The fixed vector it was built on did not.