Barratt and Sharma drove the Inception Score to 900 of 1000 with one image per class

A classifier's verdict became the field's scoreboard

When a lab trained a generative adversarial network to make images, it proved the network had improved by reporting one number. That number was the Inception Score, introduced in 2016 by Tim Salimans, Ian Goodfellow, and their colleagues at OpenAI.¹ It stuck. Two years later, when BigGAN set a new mark for generating ImageNet pictures, its headline result was an Inception Score of 166.5, up from a previous best of 52.52.² Higher was read as better pictures.

The Inception Score reads only the generator's own output. A real photograph never enters the calculation at all.¹

What the classifier is asked about each picture

The score runs every generated image through one frozen classifier: the specific Inception network trained on ImageNet, a collection of 1.2 million photographs sorted into 1000 categories.¹ For each image, the classifier returns a label guess, a probability spread across those 1000 categories: this much goldfish, this much tabby cat, this much minivan. Call that guess for a single image p(y \mid x). Average the guesses over every image the generator makes and you get p(y), the generator's label spread across its whole output.

A high score rewards two things at once, and they line up only when the generator is good. Each single image should look like one clear thing, so the classifier is confident and piles p(y \mid x) onto a single category. Across all the images, the generator should cover many categories, so p(y) is spread out rather than stuck on one. The score measures the gap between those two distributions for each image, averages it, and then exponentiates the result, which the authors did only "so the values are easier to compare."¹

The gap itself is a KL divergence, a standard measure of how far one probability distribution sits from another: zero when the two match, larger the more they differ. A confident guess that is far from the overall average is exactly what a sharp, varied generator produces.

Because the classifier can output 1000 categories, the score runs from 1 to 1000.³ That ceiling is worth holding onto: a score of 900 means 900 out of a possible 1000, most of the way to the top.

The test that earned the number its trust

The score was not pulled from nowhere. Salimans and colleagues checked it against people. They showed Mechanical Turk workers a mix of real and generated CIFAR-10 images and asked which were real. The workers were right 78.7 percent of the time. When the generated samples were filtered down to the top one percent by Inception Score, the workers' accuracy dropped to 71.4 percent. The better the score, the harder the images were to tell from real photographs.¹

In the table that introduced the number, the real CIFAR-10 images scored highest of all, 11.24, above every generator the paper tested.¹ Within that setting, generators trained on their own objective and judged on a dataset the classifier understood, the Inception Score tracked what human eyes saw. Barratt and Sharma, who would later take the number apart, did not dispute that it worked there.³

The generator that scored 900 and made one picture per class

The reading most people reach for is that a higher Inception Score means better images. In 2018, Shane Barratt and Rishi Sharma, then at Stanford, showed it need not. They ran an optimization straight against the Inception network, nudging pixels until the classifier grew more confident and its category spread widened, cycling through all 1000 target categories so the result was one tuned image per category. Starting from random noise, the images reached an Inception Score of 986.10, a hair under the maximum of 1000, while looking like colored static.³

That first result a defender can set aside. The score's own authors had warned of it, and had said plainly that the number was only ever a rough guide for models trained on a separate objective.¹ Nobody ships optimized static.

The second result is harder to wave away. Starting instead from the output of a GAN trained on CIFAR-10, the same procedure reached about 900, with images the authors described as realistic-looking.³ A generator making recognizable pictures, one per category, sat within a tenth of a perfect grade.

One picture per category is all the score asks for. It is satisfied when p(y) spreads across the 1000 categories and each image is classified confidently, and a single strong image per category does both. Whether the generator can produce a thousand different goldfish or only ever the same one goldfish makes no difference to the number, because the spread is measured across categories and never within them.³ A generator that had collapsed to one fixed image per category, the narrowest output imaginable, would still post a near-perfect score. Mode collapse, the failure where a generator keeps emitting the same few images instead of the variety in its training data, is the thing GAN researchers most wanted a number to catch. The Inception Score could not see it.

The number also moved for reasons that had nothing to do with the pictures. Three implementations of the Inception classifier with nearly equal ImageNet accuracy scored the exact same CIFAR images 11.5 percent apart, and a routine batching setting shifted a score by close to a point.³ Point the classifier at a dataset it was not trained on and it misreads everything: on CIFAR-10, the categories it most often assigned were labels like Moving Van, Sorrel, and Threshing Machine, none of which CIFAR contains.³

Why the field stopped ranking on a single number

The cost was years of reported progress resting on a number that could be gamed and could not see the failure researchers cared about most. A later large study drove the point home from the other side: with enough hyperparameter search and random restarts, competing GAN algorithms reached similar scores, so a reported gain could reflect tuning and compute rather than a better model.⁴

The deeper problem is not special to this score. A single number cannot separate two different failures. Sajjadi and colleagues argued that a one-dimensional score cannot tell a generator that makes poor images apart from one that covers only part of the target, because sample quality and coverage are distinct things to measure.⁵ Theis and colleagues had shown, even before the Inception Score existed, that doing well on one evaluation criterion need not mean doing well on another, so no single score stands in for overall quality.⁶ By 2018 there was no agreement on which measure to trust. One review counted more than two dozen.⁷

The number that took its place closed the specific hole. The Fréchet Inception Distance compares the statistics of generated images against those of real images, the comparison the Inception Score never makes. When researchers degraded images step by step, FID climbed steadily with the damage while the Inception Score stayed flat or even fell.⁸ FID is not free of the one-number problem, since it too returns a single scalar. But it looks at the real images, which the Inception Score never did.