COMET scores a translation by predicting the grade a human would give it

The count that ran the field lost to a model

For a decade, almost every machine-translation system was judged by one number. Of 769 translation papers published at the main computational-linguistics conferences between 2010 and 2020, 98.8 percent reported BLEU, and 74.3 percent reported nothing else.¹ BLEU is a count: it measures how many short runs of words a system's output shares with a human translation of the same sentence, and it never models what either sentence means.

At the 2022 Conference on Machine Translation, the organizers ranked 13 such metrics against the judgments of professional translators. BLEU came last, with an average rank of 5.31. First, at 1.32, was a metric called COMET.² COMET is not a counting procedure. It is a neural network, trained to predict the score a human would have given a translation.³ In the largest comparison of its kind, run against 2.3 million human judgments across 232 language directions, COMET agreed with human rankings on 83.4 percent of system pairs where BLEU managed 74.6, and the authors recommended dropping BLEU entirely.⁴

A COMET score is a guess at a human judgment, produced by a model that learned what a good translation looks like from a particular pile of human ratings. Where those ratings were rich, COMET is sharp. Where they were thin, it is blind.

The human scores COMET copies

COMET learns from two kinds of human judgment, and it can only be as discerning as they were. The first is Direct Assessment. A person reads a translation and drags a slider from 0 to 100 to rate how completely it carries the meaning of the original. The ratings come from crowd workers, standardized per rater so one person's harsh 40 counts the same as another's.⁵ It is a single number for the whole sentence, with no record of what was right or wrong in it.

The second is MQM, or Multidimensional Quality Metrics. Here a professional translator marks each error in the output, tags it by type, and grades its severity. The score is a weighted sum of those errors: a minor error costs 1 point, a major error 5, and a complete non-translation 25.⁶ MQM records exactly what went wrong; Direct Assessment records only how a rater felt about the whole.

The two do not agree. When professional translators re-scored systems with MQM that crowd workers had rated with Direct Assessment, the two methods produced sharply different rankings of the same systems, and the crowd's judgments correlated poorly with the experts'.⁶ COMET's early models learned mostly from crowd Direct Assessment. Whatever the crowd failed to notice, the model was trained not to notice either.

Turning three sentences into one number

To convert those ratings into a metric, COMET starts from a pretrained multilingual model, a network already trained on large amounts of text in many languages. It turns each sentence into a list of numbers that stand in for its meaning. COMET's founding version used one called XLM-RoBERTa.³ The rest of the procedure is short.

1. Read three sentences

   The original, the system's translation (the candidate), and a human translation of the original (the reference). Encode each into its list of numbers.³

2. Compare them

   Combine those lists into a single summary of how closely the candidate matches the meaning carried by the source and the reference.³

3. Predict a score

   Pass that summary through a small scoring layer, fine-tuned (its internal settings nudged) until the numbers it returns match the human Direct Assessment and MQM scores as closely as possible.³

A second version drops the reference and scores the candidate against the source alone. This is quality estimation, or QE: a COMET score for a translation whose correct answer was never shown to the model. It is a separate network from the reference-based one, not a setting on the same model.⁷

A COMET number moves when the translation doesn't

Suppose you read that a system scores 0.85 on COMET. On its own that number is close to meaningless.

Which model produced it? The people who build COMET ship several. Their default reference-based model returns a score from 0 to 1, where 1 is a perfect translation. But models released before 2022 return a raw number that, in the developers' own words, "does not have a direct interpretation"; the rescaling to a 0-to-1 range only arrived in 2022.⁷ A 0.85 from a 2021 model and a 0.85 from a 2022 model are not the same measurement. Was a reference used, or was it the reference-free QE model, a different network trained on different data? Which language direction produced it? The average human score behind these models runs from about 0.51 in some directions to 0.91 in others across the data COMET is trained and tested on, so a number is high or low only relative to its direction.⁸

The score moves even when nothing about the translation changes at all. Researchers ran a single fixed model on the same English-to-German test set under two releases of the COMET software and got two answers.

Same model, same translations, same test set, two software releases, and the score shifts by 0.04 with none of the translation changed.⁸ None of these facts can be recovered from the number itself, which may be why at least 12 percent of the papers one survey examined reported a COMET score without saying which model produced it.⁸ A COMET score reported without its model version, its reference mode, and its language direction cannot honestly be set beside another.

Optimizing toward COMET selects the wrong number

A metric can be used two ways. As a judge it scores translations after the fact. As a target it picks them: instead of taking a system's single best guess, you generate many candidate translations and keep the one the metric scores highest. This second use, a form of what is called minimum Bayes risk decoding, is where COMET's training shows its cost.

Amrhein and Sennrich selected translations this way and checked what COMET chose. The outputs it preferred carried wrong numbers and mistranslated names, because COMET is far less sensitive to those errors than to ordinary words.⁹ They measured that insensitivity directly, by changing one thing in a translation and watching how far the score fell.

Corrupting an ordinary noun costs six times what corrupting a number does. So when COMET sorts a pool of candidates, a swapped digit is cheap and easily outweighed. In one of their examples the source and the human reference both say a musician left his band in 1970; the translation COMET scored highest says 1980. In another a person's name is corrupted from "Tebboune" to "Tebboene".⁹ The date is wrong and the name is wrong, and COMET rated the output near the top, because a swapped digit barely moves a score built to predict a human rating in which digits rarely decided anything.

It would be easy to read this as a sign that COMET is the weaker metric. The rankings say the opposite: against professional human judgment it placed first and BLEU last, and across millions of judgments it tracked human preference more closely than any count.²⁴ The blind spot appears only when COMET stops being the judge and becomes the target. Nor is it that COMET cannot see a wrong number at all: shown a clean pair of translations that differ in a single digit, it usually prefers the correct one.⁹ The failure needs a realistic pool, where its mild dislike of a wrong number is swamped by other differences and a meaning-breaking output wins. And newer did not mean safer here: the checkpoint trained on MQM, the more careful of the two human signals, chose a wrong number in 16.6 percent of cases against the older Direct-Assessment model's 7.2 percent.⁹

The trap is general. Push any system hard enough toward a metric and it finds the metric's blind spots. The same survey that traced COMET's version problems found it assigning positive scores to empty and wrong-language outputs. At a 2024 competition, a translation system used this decoding to dominate the automatic metrics without matching that lead under human evaluation.⁸