The system human judges ranked best at the 2005 NIST evaluation scored 6th on BLEU

A score built from one sentence pair

The paper this course already read reported a Transformer that scored 28.4 BLEU translating English into German, more than 2 points above every prior system, ensembles included.¹ Read alone, 28.4 says almost nothing. BLEU is not a percentage of sentences judged correct. It is the output of one counting procedure, published in 2002 by four IBM researchers: Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.² Run that procedure once by hand and every BLEU score afterward stops being a black box.

BLEU compares two sentences: a candidate, the output a translation system produced, and a reference, a translation a human wrote for the same source text. Those are the paper's own terms, reused exactly here.² Hold one pair fixed for the rest of this lesson.

Candidate: "the small brown dog chased the red ball across the yard." Reference: "the small brown dog quickly chased the red ball across the yard." The candidate is the reference with one word removed, "quickly." Every other word, in the same order, is identical: grammatically fine, missing one word's worth of meaning.

Counting runs of words, then capping the count

BLEU starts by counting how many short runs of consecutive words the candidate shares with the reference: runs of one word, two, three, four. A run of two words is a bigram, three a trigram, and the general name for a run of n words is an n-gram. Every one of the candidate's 11 unigrams appears somewhere in the reference, since the reference contains the same 11 words plus "quickly." A first pass at scoring might just divide matches by total candidate n-grams and call it precision.

That naive count breaks on a case the authors used to justify a fix: a candidate reading "the the the the the the the," scored against a reference containing "the" only twice.² Counted naively, all seven repeated words match, for a precision of 7 out of 7. The fix caps each word's count at how many times that word appears in the reference, so "the" can contribute at most 2 matches no matter how many times the candidate repeats it. Papineni's paper calls that cap clipping.²

Clipping changes nothing in the dog-and-ball example, since no candidate word repeats more often than it appears in the reference. What it changes is where matches fail: only at the exact point "quickly" was removed.

Precision falls every time the window widens: a single dropped word costs nothing at the 1-word view and a quarter of the 4-word view. BLEU never asks whether the word mattered, only how many fixed-length windows it broke.

Two steps remain.

1. Average the four precisions

   Not by adding and dividing, but by geometric mean: multiply all four together and take the fourth root. 1.000 × 0.900 × 0.778 × 0.625 = 0.4375, and its fourth root is about 0.813. A single missed word already costs the candidate almost 19 percentage points, from n-gram counting alone.²

2. Apply the brevity penalty

   Precision alone can be gamed by saying less: a candidate that stopped after "the" would score 1.000 on the one word it chose. Papineni's fix multiplies the score by a brevity penalty whenever the candidate is shorter than the reference.² Here the candidate is 11 words, the reference 12, so BP = e^(1−12/11) = e^(−1/11) ≈ 0.913. Being one word short costs another 7.1 percentage points, from 0.813 down to the final score below.

3. Multiply the two together

   BLEU is the brevity penalty times the geometric mean of the precisions: 0.913 × 0.813 ≈ 0.742, or 74 on the common 0-to-100 scale. A candidate that is grammatically perfect and missing one adverb's worth of meaning already lands inside the range Google's own practitioner guidance calls "quality often better than human," above 60 points.³

What one score does and does not settle

The paper that defines BLEU states its own limit plainly: "BLEU only needs to match human judgment when averaged over a test corpus; scores on individual sentences will often vary from human judgments."² BLEU is a corpus-level statistic. Add up matches across a whole test set, and the noise in any single sentence washes out. Quote a BLEU score for one sentence, as the dog-and-ball score above was, and Papineni's own paper says not to trust it alone.

A subtler problem sits underneath a corpus-level score: BLEU is not one metric but a family of configurations, and a bare number does not say which one produced it. Matt Post, who built the reproducibility tool sacreBLEU, names the hidden choices: reference count, brevity-penalty handling, longest n-gram length, and tokenization.⁴ Each choice moves the number without moving the translation.

Two labs both reporting "BLEU 34" on the same test set can have measured different things. Post's own survey of well-cited MT papers found that even the Transformer paper this lesson opened on does not state, in a checkable way, which tokenization produced its 28.4.⁴ SacreBLEU closes that gap by printing a signature string alongside every score, such as BLEU+c.mixed+l.en-de+#.1+s.exp+t.wmt14+tok.13a+v.1.2.10, recording the reference count, tokenizer, and dataset version behind the number.⁵

Correlation with human judgment is a second question, above tokenization. Papineni's original paper reported 0.99 and 0.96 with two judge groups, from 5 systems, one language pair, 250 rated pairs, run once.² A structured review of 34 later papers found a wider picture: system-level BLEU correlation for MT is "mostly Medium or High," on a scale where High means 0.85 or above.⁶ The same review's year-by-year table for German-English shows why that needs qualifying. Correlation for one direction ranges from 0.12 to 0.90 across ten WMT years; for the other, from −0.43 to 0.83.⁶

The evaluation where the ranking flipped

In 2005, NIST ran an Arabic-English machine translation evaluation with seven entries. Six were phrase-based statistical systems trained on the same parallel corpus, most tuned by directly maximizing BLEU on their own training data.⁸ The seventh was a hybrid: monolingual English speakers with no Arabic selecting among alternative machine-generated translations of phrases from the source, then post-editing the result.⁸ Human judges ranked that hybrid entry 1st. BLEU ranked it 6th.⁸

Callison-Burch, Osborne, and Koehn traced the gap to that one system. Correlate the seven entries' BLEU scores against human adequacy ratings and R² comes out to 0.14, barely any relationship. Drop the hybrid entry and R² jumps to 0.87; fluency correlation moves from 0.002 to 0.742 under the same removal.⁸ One system alone accounted for nearly all of BLEU's disagreement with the people reading the translations.

A sentence pair from that evaluation shows why. One hypothesis matched more reference words overall; a second matched fewer.

Human judges scored the hypothesis with fewer total matches higher on both adequacy and fluency.⁸

Word order compounds the problem. A candidate with k words and b matching bigrams has (k−b)! ways to permute around every bigram break without changing the BLEU score. In the actual 2005 NIST data, that count exceeds 10⁷³ for some sentences.⁸

Callison-Burch, Osborne, and Koehn drew a narrow conclusion: BLEU works for tracking incremental change within one system, and for comparing systems built on similar strategies. It fails across different strategies, on gains it was never built to model, and on gains that surface rarely in a test corpus.⁸ The 2005 field had spent years optimizing statistical systems directly against BLEU. The one entry that broke from that strategy, built by people rather than tuned against the metric, was the one BLEU ranked worst.