T5 recast every language task as text in, text out, and lost at translation

One model for every language job

In October 2019 nine researchers at Google posted a preprint titled "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer."² The 67-page paper was published in the Journal of Machine Learning Research the following year.¹ The model it introduced, T5, has a plain name once you unpack it: the Text-to-Text Transfer Transformer. Transfer learning was the standard approach by then. Train one model on an enormous pile of generic text so it picks up the shape of the language, then adapt that same model to each specific task you care about. The pretraining is the expensive part you do once. The adapting, called fine-tuning, is cheap and you do it per task.

T5's contribution was to make the "each specific task" part uniform. Every task becomes the same kind of problem: text goes in, text comes out. Casting the field's benchmarks this way and combining the best training choices with scale, the model reached the best published result of its time on 18 of the 24 task scores the authors measured.¹ That number is why the paper became a reference. What it is cited to have settled is a larger claim.

Text in, text out, even when the answer is a number

Translation is the easy case. You feed the model the sentence with a short instruction glued to the front, translate English to German:, and it produces the German. Nothing surprising there. Translation was always text in, text out. The move T5 makes is to force every other task into that same mold, by changing only the words in front.

Ask it to judge whether a sentence is grammatical, and the input is cola sentence: followed by the sentence. The output is the literal word acceptable or not acceptable, not a 1 or a 0. Ask it to summarize, and the prefix is summarize: and the output is the shorter text. The paper puts all four in the diagram on its first page.¹

The third row is the odd one out. That task, STS-B, scores how similar two sentences are on a scale from 1 to 5, so the right answer is a number like 3.8, not a category and not a sentence. A model that only ever emits text cannot hand back a number the way a calculator would. T5's answer is to treat the number as text too. It rounds the target to the nearest 0.2 and prints the digits as a string, so a score of 2.57 is trained as the characters 2.6.¹ Even the task that fits the interface least is made to fit, at the cost of a little accuracy.

The payoff of one interface is that one loss and one decoding procedure train and run all of it. There is no per-task output layer to design, no bespoke head bolted on for classification and another for regression. When you load T5 from a modern library today, you still address it by writing the same task prefixes in front of your text.³

The 750 gigabytes T5 read to learn English

A model that learns language from generic text needs a great deal of it, and it needs that text readable rather than raw web sludge. The authors built their own corpus for the job and called it C4, which the released dataset card describes as "a colossal, cleaned version of Common Crawl's web crawl corpus."⁴ Common Crawl is a public scrape of the web that produces about 20 terabytes of text a month. Almost all of it is unusable as training data: menus, error messages, boilerplate, and code.¹

What made C4 usable was a short list of blunt filters. Keep only lines that end in a real sentence-ending mark. Throw away any page with fewer than five sentences. Drop any page that contains a curly bracket, which cheaply removes most code. Drop pages with "lorem ipsum" placeholder text. Remove any three-sentence span that appears more than once, which strips repeated boilerplate. Keep only pages a language detector calls English with at least 99 percent confidence.¹ The cleaning is severe: it discards roughly seven-eighths of the raw crawl.⁴

Here the paper and the download part ways. The paper reports its C4 at about 750 gigabytes.¹ The version you can fetch today was regenerated later. It measures 806.87 gibibytes of cleaned English, about 365 million training examples, cut down from 6.21 tebibytes of uncleaned crawl.⁴ Same recipe, different run. A figure attached to T5 belongs either to the paper's original run or to an artifact released later, and the two do not always match.

A bake-off, then scale poured on the winner

The core of the paper is not the final model but the method that produced it. The authors fixed a baseline, then changed one ingredient at a time, holding everything else constant, and measured what each change did. Which training objective works best. Which architecture. How much data. Because only one thing moves at a time, a change in the score can be pinned to that one thing, not to a lucky combination. This is a controlled experiment.¹

One of those comparisons is the architecture question, and its answer is the one T5 still gets cited to settle. The original encoder-decoder form, the shape with a stack to read the input and a separate stack to write the output, worked best inside their text-to-text framework. It uses about twice the parameters of an encoder-only model like BERT or a decoder-only language model, but at similar computational cost, and it won.¹

Then the winning choices were combined and scaled. The paper does not report a single headline compute figure in the flops-and-days form sometimes quoted, so the scale is given in tokens instead, the word-and-word-piece chunks a model reads. The baseline was pretrained on about 34 billion tokens. For comparison, BERT had seen roughly 137 billion and RoBERTa about 2.2 trillion. The final T5 models were trained on about a trillion tokens, roughly 32 times the baseline's diet.¹ How many tokens a model of a given size should be fed is a question later scaling work took up directly. T5's number is simply the budget these authors chose.

Size was also scaled, across five models from about 60 million parameters up to 11 billion. The largest has an unusual shape. Its internal width is a modest 1,024, but its feed-forward layer is widened to 65,536, with 128 attention heads. Very wide layers run efficiently on the accelerators used to train it, and the released 11B configuration carries exactly those numbers.⁵ One caution on the checkpoints. The paper's original models and the improved t5.1.1 checkpoints released later are not numerically identical. The smallest released model reports a slightly different parameter count than the paper's does.⁶

What it showed, and what it is asked to have settled

T5's 18-of-24 result is real. On GLUE, a suite of language-understanding tests, the 11B model scored 90.3 against a previous best of 89.4. On the harder SuperGLUE it reached 88.9 against 84.6, and on the SQuAD reading test its F1 was 96.22 against 95.5.¹ But the same table records where it lost, and it lost on every task of one kind.

The WMT rows are the three translation tasks, scored in BLEU, a measure of how closely the output matches a reference translation. T5 came in below the previous best on all three, and by a wide margin on the low-resource English-to-Romanian task, 28.1 against 38.5.

The authors trace the shortfall to their own setup. They pretrained on English only, while the translation systems they were up against used backtranslation, a technique that manufactures extra training pairs. A model that framed the whole field as text-to-text was weakest at the one task that is most literally text in, text out, because that task rewards choices T5 did not make.¹

Two more things have aged. The scores are frozen to their moment. Every "previous best" in the table is dated as of 24 October 2019, and T5's own numbers have since been passed. On some reading tests the paper already scores above the human baselines it cites.¹ The encoder-decoder finding held only at the matched, modest scale the bake-off ran at. The mainstream of large language models since has consolidated on the decoder-only design that comparison passed over, the GPT-3 direction. The paper's controlled comparison did not settle the architecture question for the far larger scale the field went on to reach.

Even the way T5 is used moved past the paper. As shipped in 2020, it was fine-tuned per task. A 2022 line of work added instruction fine-tuning on top of the T5 family, producing the released Flan-T5 checkpoints, which their authors report reach strong few-shot performance against much larger models. That is a capability the original paper did not study.⁷