A high F1 score can hide a model that missed an entire class

One number from four counts

The F1 score is a single number between 0 and 1, and it decides a large share of public claims about classifiers. Those are the systems that sort email into spam and not-spam, scans into tumor and clear, documents into relevant and not. Reading-comprehension benchmarks report it, and so does almost any paper announcing a state-of-the-art classifier. The number is old. It grew out of information retrieval, where C. J. van Rijsbergen defined the two quantities it combines and built a measure to weigh them together.¹ Both quantities, and F1 itself, come from four counts you can read off a small table.

Precision and recall, from a table of four

Set a classifier loose on a batch and sort its verdicts against the truth. Every item lands in one of four boxes. It was positive and the system called it positive: a true positive. It was positive and the system missed it: a false negative. It was negative and the system falsely flagged it: a false positive. It was negative and the system correctly left it alone: a true negative. Take a search that returns 10 documents from a collection of 1,000, when 12 of those documents are actually relevant.

Precision reads down the column of things the system called positive: of the 10 it returned, 8 were relevant, so precision is 0.8. Van Rijsbergen defined it as the chance that a returned item is relevant.¹ Recall reads across the row of things that were actually positive: of the 12 relevant documents the system found 8, so recall is about 0.67, the chance that a relevant item gets returned.¹ In counts, that is \mathrm{precision} = \mathrm{TP}/(\mathrm{TP}+\mathrm{FP}) and \mathrm{recall} = \mathrm{TP}/(\mathrm{TP}+\mathrm{FN}).² Neither precision nor recall uses the box holding 986.

Why the mean is harmonic

Two numbers invite an average, and the obvious average is the arithmetic mean. F1 does not use it. F1 is the harmonic mean of precision and recall,³ and the harmonic mean has one property that matters here: it sits close to the smaller of the two numbers rather than halfway between them.

Watch it on a lopsided pair. A fingerprint system with precision 1.0 and recall 0.2 is almost useless. It is right whenever it speaks, but it stays silent on four cases in five. The arithmetic mean rewards it with 0.6. The harmonic mean gives it 0.33, which is about what a nearly useless system deserves.³ A lopsided pair cannot buy back a low recall with a high precision.

The measure did not arrive with the name F. Van Rijsbergen wrote it as an effectiveness score to be minimized, fitted with a single dial that lets a user weight recall against precision.¹ The evaluation community adopted it at the fourth Message Understanding Conference in 1992, wrote it in the form still used, and called it the F-measure. Its parameter β sets recall's importance relative to precision. β above 1 weights recall more, β below 1 weights precision more, and β equal to 1 gives the equal-weight case everyone now writes as F1.⁴ The letter itself was a mild accident, picked from a different F in van Rijsbergen's book.³

The cell F1 never reads

Look back at the four boxes. F1 uses three of them. True positives, false positives, and false negatives all appear in precision or recall. The fourth box, the true negatives, never enters the formula.⁵ In the search example that box held 986 documents, and F1 would read exactly the same if it held 986,000. David Powers put the general point plainly: the precision-recall-F family ignores how well a system handles the negatives, so a system that is genuinely worse can score higher.⁵

Three consequences follow, and each is a question to ask of any F1 you are handed.

Which class is positive? F1 is not symmetric. Swap the labels for positive and negative and the score changes, so the same predictions earn two different F1s depending on which class you decided to count.²

At what base rate? Because F1 turns on the positives, its best operating point moves with how common the positives are, and two F1 scores from datasets of different prevalence are not comparable.⁶ Push that to the limit and it gets worse: a classifier that knows nothing at all maximizes its F1 by labeling every single item positive.⁶ Recall climbs to 1, precision settles at the base rate, and the harmonic mean of the two can clear what an honest but cautious classifier manages.

Which averaging? F1 is defined for a two-class problem, so scoring three or more classes means averaging the per-class F1s, and there is more than one way to do it. Micro averaging pools the counts across all classes and takes one quotient; with every class included, that number is simply accuracy.⁷ Macro averaging takes the plain mean of the per-class scores, giving a rare class the same weight as a common one.⁷ Weighted averaging weights each class by its size and can land outside the range between precision and recall.⁷ Same predictions, three different numbers.

F1 crowns the wrong classifier

Davide Chicco and Giuseppe Jurman built the cleanest demonstration of the cost. They ran classifiers on skewed data and set F1 and accuracy against the Matthews correlation coefficient, or MCC. MCC uses all four boxes and reads as a correlation between prediction and truth, running from −1 through 0, a coin toss, up to +1.²

In the first case the data is 91% positive, and a classifier calls almost everything positive: it catches 90 of 91 positives and 0 of 9 negatives. Its F1 is 0.95 and its accuracy is 0.90, both excellent. Its MCC is −0.03, the score of a coin toss, because it never once got a negative right.² The second case is balanced 50-50, and there F1 alone is fooled. It reads a passable 0.66 while the classifier finds only 5 of 50 negatives, and both accuracy at 0.52 and MCC at +0.07 report the coin toss that it is.²

The same reversal shows up on real data. On a colon-cancer gene-expression dataset of 62 patients, ranking classifiers by F1 crowns a k-nearest-neighbors model at 0.87, ahead of gradient boosting at 0.81. MCC reverses them: gradient boosting leads at +0.55 to k-NN's +0.48. The k-NN model wins on F1 by finding cancer well, catching 92% of the sick. But it clears only 52% of the healthy, barely better than a coin toss on the patients it should be reassuring.² Rank by F1 and you ship the model that sends nearly half of the healthy patients on for more tests.

Even inside multi-class F1 the ranking is not safe. Two formulas both travel under the name macro F1: the plain mean of the per-class F1s, which scikit-learn computes by default, and the harmonic mean of the averaged precision and averaged recall, which some shared tasks use. Juri Opitz and Sebastian Burst showed the two can disagree by as much as 0.5 and, worse, can order two classifiers in opposite directions. On one pair of confusion matrices, moving from the first to the second raises the second formula from 0.50 to 0.55, an apparent improvement. The first formula does the opposite, falling from 0.49 to 0.48.⁸ A macro-F1 leaderboard has to say which macro F1 it means.

None of this makes F1 a bad measure. Van Rijsbergen built it for document retrieval, where the true negatives are every document you correctly ignored, an unbounded and uninformative mass.¹ There, accuracy is the useless number. Label every document irrelevant and you score near-perfect while returning nothing, the same way Chicco's majority classifier scores the majority fraction by doing nothing.² In a needle-in-a-haystack search, ignoring the true negatives is exactly right, because those negatives carry no news. Powers calls the same blindness to true negatives a bias, and whether it is a feature or a bias depends only on whether the negatives are worth counting.⁵ MCC is no free lunch either: it is undefined when a whole row or column of the table is empty, and Chicco resolves those cases only by convention.²