The 2009 ImageNet paper never measured the accuracy it is famous for

What the 2009 paper actually built

In 2009 a group at Princeton led by Jia Deng and Li Fei-Fei published a paper at the CVPR computer-vision conference with a plain title: ImageNet: A Large-Scale Hierarchical Image Database.¹ The words deep learning do not appear in it. Neither does AlexNet, nor the competition or the accuracy score that the name ImageNet now calls to mind. What the paper describes is a database: a large, organized collection of labeled photographs, built so that a computer-vision researcher could train and test programs on real images at a scale nobody had assembled before.¹

The construction has three moves. Start with a ready-made map of concepts. Fill each concept with candidate pictures gathered from web image search. Then pay people to throw out the pictures that do not belong.

Twelve subtrees and 3.2 million images

The map of concepts came from WordNet, a lexical database of English built at Princeton in the 1990s. WordNet groups words into synsets: sets of synonyms that each name a single concept, so that {car, auto, automobile} is one synset and {cat, true cat} another.² The synsets are linked by an IS-A relation: a bed is a kind of furniture, a Burmese cat is a kind of cat. That relation turns the vocabulary into a tree that climbs to a single root.² ImageNet borrowed that tree. Every node it wanted to illustrate was a synset, and the job was to hang real photographs under it. WordNet offered roughly 80,000 noun synsets to aim at.¹

By the time the paper went to print, ImageNet had filled a corner of that tree. It held 12 subtrees, among them mammal, bird, vehicle, furniture, and musical instrument, with 5,247 synsets and 3.2 million images in all, an average of more than 600 images per concept.¹ The paper was candid that this was a start. Its stated goal was to reach roughly 50,000 synsets and 50 million images within about two years, enough to fill the majority of WordNet's noun synsets.¹

It did not get there on that schedule. The database ImageNet eventually became, the version researchers now download as ImageNet-21K, holds about 14.2 million images across 21,841 categories.³ That is far larger than the 2009 snapshot and still well short of the 2009 projection.

How a crowd vote became a trusted label

Gathering candidate images was the easy half. ImageNet queried commercial image search engines with the words in each synset, widened the net with parent terms, and ran the queries in Chinese, Spanish, Dutch, and Italian as well as English.¹ That returned a flood. After removing duplicates, each synset started with more than 10,000 candidate images. But image search is a coarse tool: only about one candidate in ten actually showed the thing it was supposed to.¹ Ninety percent was noise, and the whole value of the database depended on getting it out.

So ImageNet sent every candidate to Amazon Mechanical Turk, an online marketplace where anyone can pay strangers small sums to do small tasks, and asked workers a single question: does this image contain the concept? The trouble is that people disagree. Show several workers a set of images labeled Burmese cat and some will click yes on a picture the others reject, because most people cannot tell a Burmese cat from any other brown cat.¹ One worker's answer cannot be trusted, and neither can a bare majority when the category is hard.

The paper's answer, and its real contribution, was to let the number of votes a label needs depend on the category. For each synset it first drew a small sample of images and had at least 10 workers vote on each one. Those votes built a confidence table: given, say, two yes votes and no dissent, how sure can we be that an image belongs? For a broad category like cat, two agreeing votes are plenty. For Burmese cat the same two votes leave real doubt, so the system keeps collecting votes until it reaches a confidence threshold set for that category, then labels the remaining candidates to that standard.¹ An image is kept only when it clears the bar.

How well did it work? The authors handed a random sample of 80 synsets, drawn from across the mammal and vehicle subtrees, to a separate group of people to re-check. On that sample, 99.7% of the images ImageNet had accepted genuinely contained their concept.¹ That figure measures one specific thing: whether a kept image really shows what its label says. It is not a claim about anything a later program would go on to do with the pictures.

The 2012 jump belongs to a different paper

None of this is why most people have heard of ImageNet. The story usually told is that ImageNet showed the world that data, not just clever algorithms, could drive machine recognition, and the evidence offered for it is a sudden leap in accuracy. That leap is real. It is just not in the 2009 paper, and it did not happen for another three years.

It happened in a competition. Starting in 2010, the ImageNet team ran an annual contest, the ImageNet Large Scale Visual Recognition Challenge, on a fixed slice of the database: 1,000 categories with about 1.2 million training images.⁴ The contest is where models were scored, using top-5 accuracy, which counts a guess correct if the right label is among the model's five best answers. In 2012 a neural network called AlexNet won that contest with a top-5 error rate of 15.3%, against 26.2% for the next-best entry.⁵ That gap is the jump. It belongs to a later paper, a different set of authors, and a 1,000-category subset. It does not belong to the 2009 database paper, which mentions no competition and no such score.

It would be wrong, though, to say the 2009 paper had nothing to say about the value of data. It ran its own small experiments showing that cleaner labels and more images per category improved a simple recognition method, and it pitched ImageNet plainly as a resource for training and benchmarking such methods.¹ The idea that more good data helps was already on the page in 2009. What arrived in 2012 was a method strong enough to turn data at that scale into a headline result, not the idea itself.

One more thing is not on the page: the figures people quote about how ImageNet was built. The often-repeated counts (49,000 workers from 167 countries sorting more than 160 million candidate images) come from later retrospectives, not the 2009 document, which gives no worker total at all.³ Its only head count is the at-least-10 workers who voted on each image of those initial samples.¹

The labels were never pristine

ImageNet's record for accuracy hardened into an assumption that its labels were essentially perfect. Two later studies, both able to examine the finished database in ways the 2009 team could not, complicate that.

In 2021 a group auditing popular benchmark test sets estimated that at least 6% of the labels in the ImageNet competition's validation set are wrong.⁶ This does not overturn the 99.7% from 2009, because the two numbers measure different objects. The 2009 figure asked whether a kept image contains its concept, on the mammal and vehicle synsets. The 2021 figure asks whether the single label pinned to an image in the 1,000-category competition set is the best one, where photos holding several objects and fine distinctions between breeds drive much of the error. Both can be true at once.

The second study came from ImageNet's own creators. Because the project set out to illustrate every noun in WordNet, it imported WordNet's whole vocabulary of people, including categories that are offensive or that no photograph could ever settle. Auditing the person subtree, which holds 2,832 categories and about 8.3% of all images, they flagged 1,593 as unsafe to use. Of the 1,239 that remained, they judged only 158 to name something you can actually see in a picture.⁷ The map ImageNet borrowed was never designed to be filled with photographs, and in places it should not have been.