Prompt engineering lifted CLIP's zero-shot ImageNet score by nearly five points

CLIP labeled ImageNet without ImageNet's labels

Early in 2021, a team at OpenAI led by Alec Radford and Jong Wook Kim released a vision model called CLIP and reported a result that has been repeated ever since. On ImageNet, the standard test that sorts photographs into 1,000 categories, CLIP's best model correctly labeled 76.2% of the validation images without training on a single one of ImageNet's 1.28 million labeled examples.¹

The paper called this zero-shot transfer, and the term is worth pinning down before anything else. Zero-shot means the model sorts images into categories it was never trained to recognize: you hand it the category names in plain language, and it picks the one that fits.¹ By that measure CLIP matched the accuracy of a standard supervised ResNet-50, a widely used image classifier trained the ordinary way on labeled photos, on categories CLIP itself had never been shown.¹ It learned instead from 400 million image and caption pairs pulled off the internet.¹

Learning from 400 million captions

CLIP is trained by a method called contrastive learning, and the idea is simpler than the name. Take a batch of image and caption pairs, say four images and the four captions that came with them. There are sixteen ways to match one of those images to one of those captions, and only four of the sixteen are correct. CLIP is trained to score those four true pairings highest and the twelve wrong pairings lowest.¹

It does this by turning each image and each caption into a list of numbers, called a vector. That is the same kind of representation word2vec built for words, and turning a picture into that form is its own subject, covered in reading images. The model learns to place a true image and its caption close together in that space and to push mismatched ones apart.¹ That is how zero-shot classification works in practice. To label a new image, you write each candidate category as a short caption, such as a photo of a dog or a photo of a cat, then ask which caption's vector sits closest to the image's.¹ The category names, written as captions, are all a user supplies.

None of this was a new idea, and the paper says so. It presents CLIP as a simplified version of earlier work on learning images from paired text, and names a 2020 method called ConVIRT as its direct predecessor.¹ What CLIP added was scale. Its training set, which the authors call WebImageText, held 400 million image and caption pairs gathered from the internet, built by searching for about 500,000 terms and keeping up to 20,000 pairs for each.¹ Training the largest models took real hardware: one ResNet version ran for 18 days across 592 high-end GPUs, and the largest vision-transformer version ran 12 days on 256 of them.¹

OpenAI released the code and the trained weights, so anyone can download CLIP and run it.² The 400-million-pair dataset was never released.² Anyone could run CLIP, but no one outside OpenAI could rebuild the pile of web data it actually learned from.

Writing the labels as sentences added points

Return to how a user runs CLIP: by writing each category as a caption. The exact wording of that caption turns out to matter, and the paper measured how much.¹ On a ResNet-50 version of CLIP, handing the model the bare category name did worst. Wrapping the name in the template a photo of a {label}. raised ImageNet accuracy by 1.3 points.¹ Averaging the model's answers across 80 differently worded prompts, a trick the paper calls ensembling, added another 3.5 points.¹ Together, the paper reports, prompt engineering and ensembling improved ImageNet accuracy by almost five points, and it reports its headline scores with that engineering already applied.¹

The 76.2% headline is the best model, a vision transformer called ViT-L/14. The 1.3 and 3.5 point gains were measured on the smaller ResNet-50 version, so the exact sizes do not carry from one model to the other.¹ What carries over is the point: several points of a score reported as zero-shot trace to a person choosing how to phrase the question, the kind of hand-work the word is usually taken to rule out.

That template did not stay inside the paper. The a photo of a {label}. pattern is built into the standard tooling around CLIP today. Hugging Face's zero-shot image classification pipeline writes its candidate categories in exactly that form, as a photo of a cat and a photo of a dog.³

It handles shifted photos and fails at counting

Set against that, CLIP earned a strength the shorthand tends to drop. Ordinary classifiers trained on ImageNet lose accuracy sharply when the test images drift away from clean ImageNet photos, toward sketches, blurry video frames, or pictures filtered to fool a model. CLIP degraded far less. The paper reports that its zero-shot models close up to 75% of the gap between ordinary ImageNet accuracy and accuracy under these shifts.¹

That robustness came from the model, not from the prompt wording, and a later study pinned down its cause. In 2022, Fang and colleagues ran controlled experiments against five candidate explanations for the robustness. Their answer was the diversity of the training data. In their words, "the more diverse training distribution is the main cause for the robustness gains, with the other factors contributing little to no robustness."⁴ The web-scale pile of images, not the language captions or the contrastive method, was doing the work.

The paper also heads off a second objection. If CLIP's 400 million web images overlapped with the test sets, high scores could be memorized rather than transferred. The authors checked, measuring how much their data overlapped each evaluation set. The median overlap was 2.2%, and removing the duplicated images shifted overall accuracy by rarely more than 0.1%, with the largest gain they could find being 0.6% on one bird dataset.¹ By the paper's own measurement, contamination does not explain the scores.

The robustness has a sharp edge, and the paper draws it plainly. Zero-shot CLIP is strong on natural photographs and weak on tasks that are specialized, abstract, or fine-grained. It is poor at counting objects in a scene, at classifying satellite images, at reading German traffic signs, and at spotting tumors in lymph-node scans.¹

The recipe reproduced on open data

CLIP did not stay one OpenAI model. Because the method was public and only the data was missing, other groups rebuilt it on datasets they could release. In 2023 an OpenCLIP model trained on the public LAION-2B set of image-text pairs reached 80.1% zero-shot on ImageNet, past OpenAI's original figure.⁵

A companion study mapped how these models improve as they grow and found that the training data itself changes that pattern: OpenAI's models and the LAION-trained ones follow different curves as they scale.⁶ The same web-scale data that carried the robustness also shapes how the models scale. Both findings point back to the dataset, more than the objective, as what made CLIP work. In everyday use CLIP now works as an image-text embedding and a zero-shot classifier inside search and larger systems, with the a photo of a {label}. habit still attached.³