CLIPScore scores below chance when a caption's words are swapped

A number for how well a picture follows its prompt

Ask a text-to-image system for "a red cube on a blue sphere." You get back a picture, and you want to know whether it drew the prompt or something else. The honest check is to show it to people and ask, but people are slow and expensive, and a training run generates thousands of images an hour. CLIPScore is the number the field reaches for instead. It is a model's opinion of another model's output, standing in for a person's. Hand it the prompt and the picture, and it returns a grade for how well they match, with no human-written caption needed as an answer key.¹ That last property has a name, reference-free, and it is most of why the score caught on: you can run it on any generated image the moment it exists.

The opinion it reports is not its own. CLIPScore reads it out of CLIP, a model trained to tell whether an image and a caption belong together. So the grade is only ever as good as CLIP's judgment of image-text fit, and every strength and blind spot below is CLIP's, inherited whole. This is a different question from whether a generated image looks real, which FID and the Inception Score measure. CLIPScore says nothing about realism and those say nothing about the prompt. It grades the match.

One cosine, clipped and stretched

CLIP is two encoders trained together, one for images and one for text, on about 400 million image-caption pairs scraped from the web.² The training pulls each true pair's image vector and text vector close together and pushes mismatched pairs apart, until both encoders write into one shared space where nearness means "these go together." Once trained, the encoders turn any image and any caption into two vectors in that space. The closeness of two vectors is their cosine similarity: one number, near 1 when they point the same way and near 0 when they are unrelated.

CLIPScore is that cosine, lightly dressed. Hessel and coauthors, who named it in 2021, take the cosine between the prompt's text vector and the image's vector, clip anything negative up to zero, and multiply by a fixed 2.5.¹

That is the whole construction. There is a variant, RefCLIPScore, that also compares the image against human reference captions when you have them,¹ but the version people report for generated images is the reference-free one above, because generated images arrive without a reference. Everything the grade knows, it knows through that one cosine.

Checked on captions, trusted on generations

A number is only worth reporting if it agrees with the people it stands in for, so Hessel and coauthors measured that agreement. On Flickr8K-Expert, a set of real photographs with human ratings of candidate captions, CLIPScore ranked the captions the way expert raters did, at a Kendall correlation of 51.2.¹ That beat the older CIDEr (43.9) and SPICE (44.9) scores, which need reference captions. A correlation of 100 would be perfect agreement and 0 no relationship at all, so 51.2 puts the score and the raters just over halfway there. That result, fast and reference-free and closer to humans than the alternatives, is why CLIPScore spread.

But it was measured on one task in particular: captions written for real photographs, the image-captioning task the paper set out to serve. The job it now does is the reverse. A text-to-image score starts from the prompt and grades a generated image, and a survey of the field lists CLIPScore as one of the first reference-free alignment scores that everyone reached for once diffusion models arrived.³ The 51.2 traveled to the new task as if it came along with the metric. It did not. Hu and coauthors built a benchmark of generated images with human faithfulness ratings, and CLIPScore's agreement with those ratings fell to a Spearman correlation of 33.2.⁴ That is close to SPICE again, and well under the 59.7 their own metric reached. The credential that earned CLIPScore its trust was never collected on the task the trust is spent on, and where someone did collect it, the number is far lower.

Which object got which color

Go back to "a red cube on a blue sphere." Swap the colors and you have "a blue cube on a red sphere": the same words, a different picture. Telling the two prompts apart is the test of attribute binding, keeping track of which adjective belongs to which noun, and it is exactly what CLIP's text encoder tends to drop. The authors of VQAScore put it flatly: CLIP's text encoders "can notoriously act as a 'bag of words', conflating prompts such as 'the horse is eating the grass' with 'the grass is eating the horse'."⁵

The controlled measurements bear that out. On a benchmark that makes CLIP choose the right caption against a hard negative with one thing swapped, it scores about 63% when a relation is swapped and about 62% when an attribute is, both barely above the 50% a coin would get.⁶ Reorder a whole caption and it does no better, near 46% on the reordered-caption test.⁶ This is insufficient sensitivity rather than total blindness. The same authors shuffled every word in a caption and asked CLIP to retrieve the right image.⁶ Accuracy dropped from 50.3% to 34.1%, a real fall, just a much smaller one than scrambling a sentence should cause. CLIP notices word order a little. It does not weigh it the way a reader does.

A weak grader is worse than no grader when it grades below chance. Winoground pairs two images with two captions built from the same words, and a model earns a point on its group metric only by matching both correctly. Pure guessing scores about 16.7%, one in six. CLIPScore reaches 7.8%, under half of chance. VQAScore, which checks alignment by asking a question-answering model whether the image shows the prompt, reaches 46.0% on the same prompts.⁵

Word order is one inherited blind spot. The rest come with it. CLIP "struggles with more abstract and systematic tasks such as counting the number of objects in an image," its own authors report,² so a CLIPScore cannot reliably tell four apples from five. And because it learned from uncurated web pairs, it carries their biases, which the CLIPScore authors flag directly in closing.¹ A single similarity number folds all of this into one figure that looks decisive and hides which part it got wrong.

What happens when the grade becomes the goal

Those failures all come from reading a CLIPScore. A sharper one appears when a system is built to raise it, because then any gap between the score and real matching becomes something to exploit. The FuseDream authors showed the gap directly: starting from an image and taking one small adversarial step, a perturbation too slight to see, they found a near-identical picture with a much higher CLIP score and no more real relevance to the prompt.⁷ Optimize the number hard enough and the number is what you get, which the authors name a danger of overfitting to the score.⁷

The same gap opens in the gentler act of picking. Generate several images and keep the one with the highest CLIPScore, and you are selecting for CLIP's taste, not the viewer's. The Pick-a-Pic authors measured how far apart those are. On held-out pairs where real users had chosen the image they preferred, an off-the-shelf CLIP model agreed with the human choice 60.8% of the time. PickScore, a CLIP model fine-tuned on those same choices, agreed 70.5% of the time, past even the 68.0% at which human experts agreed with each other.⁸ The roughly ten-point gap between the stock CLIP score and the tuned one is the cost of selecting by CLIP's preferences: a best-of-N chosen that way is the best in CLIP's opinion, which is measurably not the reader's.