The Instruments
Judging 980 more documents moved a dense retriever above BM25
nDCG@10 is the single number that sorts retrieval on the MTEB and BEIR leaderboards, and it scores each system against whichever documents a person chose to judge.
An MLPerf number is comparable only to another in its box
Double the chip count from 256 to 512 and the same Llama 3.1 405B run goes 1.93 times as fast, a fact about how many chips you bought and not about a better one.
HELM's team stopped ranking models by mean win rate
Stanford's HELM team ranked its leaderboard by a count of head-to-head wins, then dropped the number after the same results kept reordering under other defensible choices.
COMET scores a translation by predicting the grade a human would give it
That grade is a learned guess, and the same translation earns a different one under another model version or another language direction, so a bare COMET number means little on its own.
AI forecasters beat the human crowd's Brier score by 0.002 on a subset of the questions
On the full test set behind that number, the same system actually scored worse than the human crowd.
o1 scored 74% on the 2024 AIME on one try and 83% on a vote over sixty-four
That higher number is a majority vote over many sampled tries, honest in OpenAI's own report and misleading the moment it is quoted without the compute behind it.
YOLOv3 tied RetinaNet on one mAP and trailed it by 4.8 points on another
COCO's headline detection score averages the overlap test across ten thresholds where PASCAL VOC used a single one, which is why the same two detectors, YOLOv3 and RetinaNet, tied under one and split by 4.8 points under the other.
On the same WinoGrande questions, more training lifts the score from 50% to 79%
To rebuild a commonsense test that machines were passing without the reasoning it required, its designers filtered out the questions a model could already guess, and within a year the machines were near the human score again.
A mean opinion score doesn't travel between tests
The same three synthetic voices from a 2013 listening test scored a full point lower on a five-point scale when researchers replayed the identical recordings eight years later next to newer systems, though the ranking between them never moved.
Nvidia lost $589 billion in a day, more than 100,000 times DeepSeek's training bill
DeepSeek's report priced one training run at $5.576 million, and investors who read that as the whole model's cost helped trigger the sell-off on 27 January 2025.
CLIPScore scores below chance when a caption's words are swapped
Text-to-image papers trust it to check prompt alignment, but the human-correlation numbers behind that trust were measured on image captioning, a different task where the score is far stronger.
BERTScore grades how much two sentences look alike to a language model
Introduced in 2020 to go past word-counting scores, it rates a candidate by how closely a pretrained encoder sees it resembling the reference, which rewards fluent wrong answers and carries the encoder's blind spots.
A high F1 score can hide a model that missed an entire class
The harmonic mean of precision and recall sits under a large share of classification leaderboards, and it never touches the one cell of the confusion matrix that would show a model ignoring the rare class.
Barratt and Sharma drove the Inception Score to 900 of 1000 with one image per class
The Inception Score grades a generator by passing its pictures through a frozen ImageNet classifier, never once comparing them to a real photograph.
OpenAI's state-of-the-art GAIA score came from the set whose answers are public
GAIA scores a general assistant on 466 real-world tasks, most of them withheld because the public ones leak too reliably to rank on.
PaLM's training ran at 46 percent efficiency, or 58, depending on which FLOPs you count
A lab reports it to show a training run used its chips well, and precision plus a sparsity footnote can swing the peak underneath it fourfold on a single card.
A sepsis model scoring 0.63 AUROC caught one case in three at the threshold hospitals used
The area under the ROC curve scores how often a model ranks a sick patient above a well one, a measure that stays silent on the threshold a hospital sets and on how rare the disease is.
OpenAI carved out MATH-500 because 4,500 of the 5,000 test problems were in its training data
A MATH score is one boxed answer matched against a key, and the numbers labs now compare under that single name run on different problem sets and different metrics.
A model that only guesses the base rate scores zero calibration error
Expected calibration error checks whether a model's stated confidence matches how often it is right, and a coarse enough set of bins can hide most of the miscalibration it is meant to catch.
LiveCodeBench scores a model only on problems released after it was trained
That date rule caught an early DeepSeek model scoring about 60 on old LeetCode problems and near zero on ones released after its cutoff. It is also why two numbers both called 'LiveCodeBench' rarely cover the same problems.
A different attack turns 1% into 88% on the same safety test
A single Cisco score branded DeepSeek R1 unsafe, and it came from 50 requests pushed by one attack.
Claude 3.7 Sonnet's 59-minute time horizon is where its success crosses one half
METR built the number from 170 timed software tasks, and its seven-month doubling, drawn past the last released model, became a public calendar for automating a month of work.
An ensemble beat SuperGLUE's human score before DeBERTa, the model credited with doing it first
DeBERTa's single model crossed the human score of 89.8 at 89.9, after Google's T5 + Meena ensemble had already cleared it at 90.2.
Two AIs reported the same 35/42 gold at the 2025 IMO, and the IMO graded only one
An IMO medal is a proof scored by hand against a rubric, which makes the grader part of the number and turns one gold medal into two different measurements.
August 2026'Gay' filled 3% of toxic comments, so the classifier marked the word toxic
The number that ranks language models for toxicity is one classifier's probability, and the same labels score African-American English as more toxic.
A model's "37% hallucination rate" was its wrong-answer share on SimpleQA
SimpleQA keeps only questions a strong model already missed and grades a declined answer apart from a wrong one, so the share it marks incorrect was never a measure of how often a model makes things up.
Adding six wrong answers to each MMLU question fixed its guessing problem
The same rebuild that expanded every question's options also swapped in enough new problems to make Math the largest subject, and two years on the top models already sit within a point of one another near 88%.
'Human parity' in speech recognition came down to how you count the humans
Word Error Rate counts the word edits between a transcript and a reference, and when IBM re-transcribed the same phone calls the human benchmark on the hard set dropped from 11.3 percent to 6.8.
A model can top the MTEB average and be ordinary at retrieval
The number that sorts the MTEB leaderboard is one flat average over 58 datasets, weighting retrieval fifteen times as heavily as summarization purely because retrieval happens to have more of them.
The same model scored 1673 or 2214 on Codeforces, depending on the scaffolding
OpenAI's own paper reports that gap, and every other AI Codeforces figure in circulation, up to o3's 2724, is the same kind of number: an estimate fit to contests the model never entered live.
The function-calling score labs cite grades one call, and usually never runs it
Berkeley's Function-Calling Leaderboard scores each model's answer against a known-correct call, so the 66.4% at the top of its board mostly reflects single calls, blind to whether the same model can chain those calls across a whole task.
At the top of the RewardBench leaderboard, higher scores predicted lower Chatbot Arena rank
AI2 built the benchmark in March 2024 to score a reward model on 2,985 fixed chosen-versus-rejected pairs. Berkeley's follow-up training work that year found the top of the leaderboard sorting the wrong way against Chatbot Arena rank.
Machines cleared SQuAD's human exact-match score without learning to read
Microsoft and Alibaba passed the exact-match line on Stanford's reading benchmark in January 2018 while still trailing humans on F1, months after a single added sentence had halved the scores of the field's best reading systems.
ImageNet lets a model guess five times, then grades it against one flawed label
The score behind AI's 'superhuman' vision dropped below the 5.1% human line by 2015, and later relabelings found nearly a third of the validation images fit more than the single label they were given.
OpenAI withheld GPT-4's BBH score, and a rival published one anyway
The 83.1% for GPT-4 in Anthropic's Claude 3 card came from Google's report, and the tasks behind it had already reached GPT-4's training data.
GPT-4o clears 61% of τ-bench tasks once and under 25% of them eight times in a row
τ-bench scores an AI agent by whether a simulated customer's conversation ends in the correct database state, while eight straight successes collapse that score.
HellaSwag's right answer is just the sentence that came next
The 2019 test built to stump machines is now solved, and much of its answer key turns out to be wrong.
Humanity's Last Exam only admits questions today's models fail
Its scores start near zero by design and climb fast, which is how a 26% run using browsing tools got read as the frontier of human knowledge.
Reversing who did what in a summary leaves its ROUGE-2 score unchanged
ROUGE grades a summary by how far its wording overlaps a human reference, a count that holds up on single-document news and goes blind to faithfulness once summaries turn abstractive.
GPT-4 answers a third of MMMU with no image at all
The benchmark's own authors rebuilt it to force models to use the image, and scores fell 17 to 27 points.
More than half of TruthfulQA is questions a model already failed
TruthfulQA's percentage is set by which questions survived a model's mistakes and by a grader fine-tuned on the authors' own labels, well before it can say anything about which models lie more.
An AlpacaEval win rate is a judge's preference, and the longer answer can win it
The judge is another language model that reads two answers to each of 805 fixed prompts and prefers the longer one often enough to move a win rate from 22.9% to 64.3% on wording alone.
FID tied two face generators that a second yardstick and human eyes told apart
The score behind "this generator makes better images" is a distance between two bell-curve summaries of what an ImageNet network sees, and you can move it a long way without changing a single picture.
A perfect needle-in-a-haystack score means the model found one planted sentence
Gemini 1.5 clears the test through a million tokens, a benchmark that rewards copying an out-of-place fact and leaves long-context reasoning untested.
A faithful summary of a false passage passes the hallucination test
Vectara's leaderboard, the number behind every 'which AI hallucinates least' ranking, scores each model on how faithfully it summarizes a passage it was handed.
The 'human-level' language score is an average of nine English tests
The line they crossed was a hurried crowd estimate, measured on tasks that leak their own answers.
DeepSeek-V3 lists 671 billion parameters and runs 37 billion per token
The count is an exact tally of a model's frozen weights, and a mixture-of-experts model can advertise one far larger than the slice that fires on any given token.
GPT-4 sits above the EU's systemic-risk line on a number OpenAI never disclosed
Training compute is the FLOP count regulators now write into law to mark the AI frontier, and almost no lab measures it: the headline figure is reconstructed after the run, two ways, each an estimate with a loose joint.
Claude Opus 5 publishes five different per-token prices
Run one fixed task on that model and its real cost swings 3.6-fold across caching, batch tiers, and the output tokens the headline number leaves out.
A ChatGPT query costs 0.3 watt-hours by one honest measure and 2.9 by another
The gap traces to four stacked assumptions, and Google's own 2025 paper shows the same query's number can swing 2.4x again depending only on what a company chooses to count.
Groq's 270 tokens a second and Anyscale's 185 are both true
The 85-token-per-second gap between them comes down to where Groq starts its stopwatch and how long the test runs, one of at least four choices that decide what any published tokens-per-second figure means.
July 2026The system human judges ranked best at the 2005 NIST evaluation scored 6th on BLEU
BLEU scores a translation by counting matching word sequences against a human reference rather than reading what it means, and a 1-2 point difference between two systems' scores is a coin flip on whether people actually judge one better.
Gemini 2.5 Pro (Experimental 03-25) fell from 92% to 87% when the exam turned over
With only 15 problems a year, one of them is worth 6.7 points of any headline AIME score.
Answer order alone swung a language model's verdict by 80 points
Judged by ChatGPT, the same two chatbot answers swung from a 2.5% win rate to an 82.5% win rate depending only on which one appeared first.
Tokenize the same sentence by letters instead of words, and its perplexity falls 66 percent
Perplexity scores a model's surprise at real text, but the same sentence scores differently depending only on the tokenizer, the blind spot that let a 2025 study catch Llama 3 favoring ungrammatical Japanese over grammatical Japanese.
A passing contamination check on GSM8K would not catch a renamed problem
GSM8K's training set went into GPT-4's pretraining before the model scored 92 percent on the test, and a separate study shows that renaming a problem's numbers alone still moves a model's score.
GPT-4's 90th-percentile bar score traces to an uncited footnote
OpenAI's "90th percentile" rests on two sentences in a co-author's footnote that name no chart and no comparison group. Recomputed against attorneys who actually passed the exam, the same 298-point score lands near the 45th percentile, and the 15th on the essay questions closest to real legal work.
Hand-writing HumanEval's 164 problems solved contamination in 2021 and caused it by 2024
Pass@k is computed from many sampled tries per problem, the same arithmetic behind every reported HumanEval score, from Codex's 28.8% in 2021 to 2024's scores above 90%.
o1 beat GPQA's recruited PhDs, not its own expert ceiling
When OpenAI's o1 scored 78 percent on GPQA-Diamond's 198 questions in September 2024 and passed a 69.7 percent human baseline, the launch was read as a machine reaching PhD level, though the validators who define that set score closer to 81.
OpenAI funded FrontierMath and had the answers before o3 scored 25% on it
Epoch AI built 350 unpublished problems that a script grades with no human in the loop, and the version of o3 that OpenAI shipped solved about 10% of them when Epoch ran the test itself.
Every context window has a second number, and it is smaller
Labs advertise 128K, 1M, even 2M tokens as the length a model will accept. A separate test measures how much of it the model actually uses, and the answer routinely lands far below the number on the box.
The same o3 scored 75.7% and 87.5% on ARC-AGI, and only the compute changed
On the same 100 held-out puzzles, the same o3 ran about $26 a task at one compute setting and about $4,560 at another — so what an ARC-AGI score means depends on the compute behind it and the eval set under it, and "AGI has arrived" was never what the number said.
Change which SWE-bench tasks are counted and GPT-4o's score doubles
SWE-bench grades a model by running a real repository's own tests in a sealed container, and OpenAI reports the same GPT-4o clearing that bar on 16% of the original 2,294 tasks but 33.2% of the 500 its annotators kept.
Chatbot Arena's Elo is a preference vote that pays out for style
The rating is a Bradley-Terry regression over anonymous head-to-head votes. The Llama 4 model Meta cited at an Elo of 1417 was a chat variant tuned for exactly what those votes reward.
The lab that ships an MMLU score rarely says how it was measured
GPT-4's headline 86.4% on MMLU shipped with its five-shot count and little else — not the prompt template, not the scoring rule, not the harness that turned 57 subjects of multiple choice into a single number.