BERTScore grades how much two sentences look alike to a language model

Meaning as a language model sees it

BLEU and ROUGE, the two scores most machine-translation and summarization papers report, both work by counting how many words a candidate sentence shares with a human reference. BERTScore, introduced in 2020 by Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Weinberger, and Yoav Artzi, grades the same candidate without counting a single shared word.¹

It now ships as a default metric in Hugging Face's evaluate library, where researchers report it beside BLEU and ROUGE.² Instead of matching words, it matches each token to its closest counterpart by meaning. A pretrained encoder, a language model like BERT, reads the whole sentence and turns every token into a vector, and BERTScore pairs each candidate token with the reference token whose vector is most similar.¹

Every token finds its closest partner

The encoder redraws a word's vector for every sentence it appears in, so "bank" beside a river and "bank" that holds money come out as different vectors. That per-sentence vector is a contextual embedding.¹ The default English encoder is roberta-large, and the library supports about 130 models in all.³

Two vectors are compared by cosine similarity: one number, between −1 and 1, for how closely they point the same way. For these length-normalized token vectors it is simply their dot product.¹ BERTScore works out that similarity for every candidate-reference token pair, then matches greedily, so each token keeps only its single most similar partner in the other sentence.¹

Recall averages those best matches over the reference:

Precision runs the same average from the candidate's side, matching each candidate token to its best reference token, and the score people report is their harmonic mean, the same F1 that combines precision and recall elsewhere.¹ An optional step, idf weighting, gives rarer reference words more weight, but the authors found it helped only sometimes and reported their main results without it,¹ and the reference implementation warns it becomes unreliable when the reference set is small.³

Two BERTScores are rarely the same measurement

Run with roberta-large, raw BERTScores sit in a narrow band, often between 0.85 and 0.95, because even unrelated sentences share many high-similarity token matches.⁴ To spread them into a readable range, the authors offer baseline rescaling: subtract a baseline, the average score over about a million random and unrelated sentence pairs, then divide by one minus that baseline.¹ On one example pair from the documentation, a raw F1 of 0.959 becomes 0.759. Across the WMT18 German-to-English set, the average F1 falls from 0.931 to 0.576.³⁴ The authors are explicit that this changes only readability: it "does not affect the ranking ability and human correlation of BERTScore."¹

A rescaled number is comparable only to another one computed the same way. Change the encoder, the layer it reads from, the library version, or even the tokenizer, and the number moves.³ Two BERTScores are the same measurement only when all of that matches. The reference implementation prints a hash of the configuration, such as roberta-large_L17_no-idf_version=0.3.0(hug_trans=2.3.0), and the authors ask everyone to publish it in their papers, a practice borrowed from sacreBLEU.³

What a high score does not promise

Because the score reads similarity out of a pretrained encoder, a wrong sentence that still resembles the reference can score high. Michael Hanna and Ondřej Bojar tested this on translation: BERTScore reliably catches obvious content-word mistakes but misses smaller meaning-changing edits when the candidate still looks like the reference.⁵ Take the reference "You're crazy, aren't you?", a correct alternate "You're crazy, right?", and a wrong candidate "You're crazy, are not you?" The content words match almost perfectly, so the broken tag question barely moves the score. Correct translations in their data averaged an F1 of 0.815, and correct translations of this function-word kind only reached 0.712, so there was little room left to mark a wrong one down. On the direct test of whether the correct sentence beats the wrong one, accuracy on function words was 42.9%, close to a coin flip, even though it ran to 75.5% overall.⁵

This sits against the paper's own robustness result, and both hold. On PAWS, an adversarial test that reorders a sentence's content words to change its meaning, BERTScore dropped only slightly while most metrics fell almost to chance.¹ PAWS reorders content words, which the matching catches. A swapped function word barely moves the vectors the encoder reads. Standing up to the first kind of change buys no protection from the second.

The blind spots are the encoder's, and they include its biases. Tianxiang Sun and colleagues scored pairs of sentences that were identical except for one demographic word.⁶ For the reference "The carpenter made a desk for the clerk because the clerk needed a desk," the candidate with "he" scored 70.14 and the one with "she" scored 38.87, a gap of 31 points on a 0-to-100 scale from a single pronoun. Across their gender set BERTScore's average gap was about 7, where BLEU and ROUGE stayed near or below 3.⁶

None of this makes BERTScore the strongest score available, either. The paper reported better correlation with human judgment than BLEU across translation and captioning, though it spread that evidence over dozens of tables rather than one figure.¹ In a large 2021 comparison of translation metrics, BERTScore ranked system pairs correctly 92.2% of the time. That put it above BLEU's 88.2%, but below the trained metrics BLEURT (93.8%) and COMET (96.5%). The authors of that study recommended COMET instead.⁷ On summarization it can fall further: when experts rated 16 systems for factual consistency, BERTScore's reported F1 correlated 0.04 with their judgments, below ROUGE-1's 0.53.⁸ That single F1 hid a split, its recall half tracking consistency at 0.66 while its precision half ran slightly negative.⁸