The Vision Transformer lost to a plain CNN until it was fed 300 million images

A Transformer built for sentences, pointed at pictures

In 2020 a team led by Alexey Dosovitskiy at Google Brain took the Transformer, the network built to read a sentence as a sequence of words, and pointed it at images with almost nothing changed.¹ The trick was to make a picture look like a sentence. They cut each image into a grid of fixed squares, sixteen pixels on a side, and turned every square into a vector, the way a language model turns a word into one. How a photo becomes a row of vectors like that is its own lesson, as is the position information that tells the model where each square sat and the self-attention the encoder runs over them. The paper is remembered for none of that. It is remembered for the accuracy it reached and the amount of data it took to get there.

The design choice that made this work is also what put the paper at risk. A 224-pixel image becomes 196 patches, a short and coarse description of a photo, and the model has no idea in advance that two patches sitting next to each other are related.¹ A convolutional network, the kind that had won every image contest for eight years, is built to know exactly that. The paper's own summary of what it found is two clauses. Trained on a mid-size set such as ImageNet "without strong regularization," the model lands "a few percentage points below ResNets of comparable size." Feed it far more data, and "the picture changes."¹ The first of those two clauses turned out to matter more than the citation history remembers.

What a convolution assumes, and a Transformer does not

To see why more data reversed the ordering, start with what a convolutional network knows before it sees a single image. Its arithmetic slides the same small filter across every position, so it assumes two things about pictures without being told. A useful pattern is local: nearby pixels belong together. And a pattern means the same thing wherever it appears, so a cat in the corner and a cat in the center are read the same way. Assumptions built into the shape of a model like that, rather than learned from examples, are its inductive bias. They happen to be true of photographs, which is why convolutions worked so well for so long.

A Transformer reading patches has almost none of that inductive bias. Every patch can attend to every other patch from the first layer, with no built-in sense that neighbors matter more or that a shape keeps its meaning when it moves. Even the position of each patch is not known in advance; the model learns it from a table of position vectors. So the Vision Transformer has to discover from data the regularities a convolution is born with. The paper compresses its whole result into one line: "large scale training trumps inductive bias."¹ With little data, the convolution's free assumptions win. With enough data, the Transformer learns assumptions of its own that are better than the ones a convolution is stuck with.

How far the picture changes

"Enough data," in this paper, means a very large jump in scale. The mid-size end is ImageNet itself: roughly 1.3 million labeled photos across 1,000 categories, the set that had defined the field's leaderboard for years.¹ The large end is JFT, a Google-internal collection the paper describes as 303 million high-resolution images across 18,000 classes.¹ JFT is private, so no one outside Google can rerun the exact experiment. The paper that first put it in the literature, by Chen Sun and colleagues, describes its own snapshot slightly differently, as around 300 million images carrying more than 375 million noisy labels.² Two papers, two snapshots of the same internal collection; keep the figures with their owners. The large set is more than two hundred times the size of the one the field had been training on.

Across that range the ordering flips, and the cleanest way to see it is to hold one model fixed while only its pretraining set changes. ViT-L/16, the "Large" configuration with sixteen-pixel patches, pretrained on the public 14-million-image ImageNet-21k, reaches 85.30% top-1 accuracy on ImageNet.¹ The same architecture pretrained on JFT reaches 87.76%.¹ Same model, same test; the only change is the size of what it saw first, and it buys 2.46 points. The largest model, ViT-H/14, pretrained on JFT, reaches 88.55%.¹ The convolutional baseline the paper measured against is BiT-L, a ResNet-152 with every layer widened fourfold, pretrained on the same JFT set, at 87.54%.³ At JFT scale the Vision Transformer edges the best convolutional network. On ImageNet alone, as the paper's own sentence conceded, it would have trailed it.

The paper made a second claim the data story tends to drop. ViT reached these numbers "while requiring substantially fewer computational resources to train" than the convolutional baselines it beat.¹ The headline was never only about how many images the model needed. It was also about how cheaply, once it had them, it turned them into accuracy.

The condition was the recipe, not the architecture

Read quickly, the finding hardened into a slogan: Vision Transformers need JFT-scale data. A peer-reviewed survey of vision Transformers, published in 2022, states the requirement as settled, that "pre-training ViT on a medium-range dataset would not give competitive results."⁴ That a survey could still state it that flatly in 2022, two years on, shows how durable the reading became. The word doing the damage is "need." A need sounds like a property of the architecture, something true of Transformers as such. What the paper showed was narrower, and it said so in the clause most citations drop. Its comparison was to ImageNet "without strong regularization," meaning without the aggressive data augmentation and training tricks that squeeze more accuracy out of a small set.

Within a year another team supplied that missing regularization. Hugo Touvron and colleagues at Facebook AI trained the same Vision Transformer architecture on ImageNet alone, with no JFT and no other outside data, using heavy augmentation. Their reference model reaches 83.1% top-1, trained on a single eight-GPU machine in a few days, before any of the further gains they get by distilling from a convolutional teacher.⁵ The JFT-scale set was not a requirement of the architecture. It was one way to buy what strong regularization could also buy. The ViT authors had named the gap, and DeiT filled it.

The other half of the slogan, that Transformers had beaten convolutions, did not survive the next idea either. Zhuang Liu and colleagues took a plain ResNet and modernized it, adopting the Transformer era's training schedule and design choices one at a time. The result, ConvNeXt, matched or beat the Swin Transformer, the leading vision Transformer of the moment, at every model size on ImageNet, and reached 87.8% with public ImageNet-22k pretraining.⁶ The 2020 gap was never convolution against attention in the abstract. It closed as soon as each side adopted the other's methods. The architecture itself kept spreading regardless: the vision half of CLIP is a Vision Transformer, trained on image-caption pairs rather than labels at all.