Reversing who did what in a summary leaves its ROUGE-2 score unchanged

The one quality number on a summarizer's card

Chin-Yew Lin defined ROUGE in a 2004 paper, giving it the name Recall-Oriented Understudy for Gisting Evaluation.¹ It grades a machine summary, called the candidate, against one or more human-written summaries of the same text, called the reference summaries. The grade is recall-oriented, meaning it asks how much of the reference the candidate managed to recover, not how much of the candidate is justified. That direction is the one design choice worth holding onto: ROUGE rewards a candidate for containing the reference's words, and stays silent on anything the candidate adds.

The BART model card is the working example. It reports a ROUGE-1 of 42.949, a ROUGE-2 of 20.815, and a ROUGE-L of 30.619 on the CNN/DailyMail news corpus.² No human rating and no faithfulness check sit beside them. To know what those numbers are worth, you have to know what they count.

Working one ROUGE score by hand

Take a reference summary of six words, "the cat sat on the mat", and a candidate that changes one word, "the cat sat on a mat". ROUGE-N counts overlapping n-grams, where an n-gram is a run of n words in a row. For ROUGE-1 the units are single words. The reference holds six of them (the word "the" twice), and five reappear in the candidate: the, cat, sat, on, mat. The candidate's "a" is not in the reference, so it earns nothing. Five recovered out of six gives a ROUGE-1 recall of 5/6, about 0.83.¹

ROUGE-2 moves to two-word runs. The reference has five bigrams: "the cat", "cat sat", "sat on", "on the", "the mat". The candidate shares the first three but breaks the last two, because "on a" and "a mat" are new pairs. Three out of five is a ROUGE-2 of 0.60. The single substituted word, which left the meaning intact, cut the two-word score from a possible 1.0 down to 0.60. ROUGE-L takes a third route: it finds the longest common subsequence, the longest run of words that appears in both in the same order but not necessarily adjacent. Here that run is "the cat sat on mat", five words long, for a ROUGE-L of 5/6 in the recall-only setting Lin used for the DUC evaluations.¹

Three counting rules, three different numbers for one pair. That spread is the first thing a reported ROUGE score hides. Yvette Graham made the point at scale in 2015. She scored the DUC-2004 summarization systems with 192 ROUGE variants. The variants differ only in the counting knobs: which n, stemming on or off, common words kept or dropped, and precision against recall against a combined score. Across all 192, their correlation with human assessment ran from 0.79 down to 0.29. The variants the field had been recommending were not the best ones, and the strongest variant, a precision-based ROUGE-2, matched BLEU rather than beating it.³ A bare "ROUGE score" on a page does not say which of those knobs produced it.

Where overlap and meaning come apart

Because ROUGE only matches strings, two candidates that share the same words score the same however differently they read. Lin built the cleanest demonstration into his own paper. Against the reference "police killed the gunman", he scored two candidates: "police kill the gunman", which is correct, and "the gunman kill police", which reverses who did what. Both share exactly one bigram with the reference, "the gunman", so both take an identical ROUGE-2 of 1/3, even though, as Lin notes, they "have very different meanings".¹ A summary that says the victim did the killing is scored no worse than one that gets it right.

ROUGE-L recovers part of what ROUGE-2 loses here, because word order feeds the longest common subsequence: the correct candidate scores 0.75 and the reversed one 0.50.¹ But the gap is a lucky by-product of surface order, not an understanding of the sentence, and the survey literature treats it that way. A 2022 survey of NLG evaluation metrics places ROUGE among the untrained, n-gram-based measures and notes that ROUGE-L "cannot differentiate between hypotheses that could have different semantic implications, as long as they have the same" matching subsequence.⁴ Meaning-blindness is not a bug a better ROUGE variant would fix. It is what a string-overlap count is.

The setting where the count earns trust

None of that makes ROUGE worthless, and it became the field standard for good reason. Lin's original tables showed ROUGE-2 correlating with human judgment as high as 0.99 on the task he tested, 100-word summaries of a single news article scored against multiple references. Its grip loosened elsewhere: on multi-document tasks the correlations "rarely reached high 90%".¹ NIST ran the DUC summarization evaluations where ROUGE was adopted, and later confirmed the pattern from the outside. At DUC 2006, ROUGE-2 correlated with human content responsiveness at 0.84 by Pearson. The agency credited the metric with "reasonable correlation with manual coverage judgments that makes it useful in system development".⁵

That home setting also carries a warning the same NIST report supplies. Because news articles put the gist up top, "simple baseline systems creating summaries from the first sentence" were hard to beat. Across the DUC years, automatic summaries "seldom performed better than simple baselines based on the structure of news articles". The references themselves generalize away from the source. One study found that no more than 55% of a reference summary's vocabulary appears in the source documents. A candidate that rewords the source just as faithfully, but in different words than the reference used, is scored down for the mismatch.⁵ ROUGE works where the summaries are short, extractive, and drawn from a single news story with a strong reference.

The dimensions overlap cannot see

When later researchers measured ROUGE against human ratings dimension by dimension, the narrowness showed up as a pattern, not a single verdict. The 2021 SummEval study collected expert ratings on four qualities for the summaries of 100 CNN/DailyMail articles and correlated each ROUGE variant against them. ROUGE came out weakest on coherence and relevance. Those are the qualities that depend on how a summary is organized and what it chooses to include. It scored higher on consistency, the agreement between a summary and the facts of its source.⁶

The consistency column is the one to read carefully. It looks like ROUGE tracking whether a summary is faithful to its source, and SummEval's own authors warn against reading it that way: the strong figure "could be attributed to the low abstractiveness of most neural models". When a model mostly copies phrasing from the article, staying close to the reference and staying faithful to the source amount to the same thing, and overlap catches faithfulness by accident.⁶

A 2020 Google study pulled the two apart on purpose. It worked on XSum, a dataset of BBC articles paired with a single abstractive sentence, so the summaries have to reword rather than copy. Human annotators rated 2,500 summaries for faithfulness, defined as every claim being entailed by the source article. Correlated against those judgments, ROUGE-1, ROUGE-2, and ROUGE-L all landed between 0.10 and 0.20 on Spearman's scale, which the authors classify as very weak, while a metric based on textual entailment reached 0.43.⁷ Where the copying stops, ROUGE's apparent grip on faithfulness goes with it.

The honest statement is that ROUGE does not track faithfulness, not that it rewards the unfaithful. The same study's best model on ROUGE, a BERT-initialized summarizer, was also its most faithful, so higher overlap and better faithfulness moved together there rather than against each other. A separate 2020 paper stated the gap flatly: "the most common metrics for assessing summarization algorithms do not account for whether summaries are factually consistent with source documents". This is against a backdrop of up to 30% of abstractive summaries carrying a factual error.⁸ A summary can be fluent, wrong, and high-scoring at once, and ROUGE has no column that would tell you.