An MLPerf number is comparable only to another in its box

How a run becomes a number

MLPerf is a set of benchmarks run by MLCommons, a nonprofit engineering consortium, and a single training result is a stopwatch reading. MLCommons fixes the model, the dataset, and a target quality, and the score is the wall-clock time a system takes to train that model from scratch until it reaches the target.¹ The clock starts when the system first touches the training data and stops when the model hits the defined quality on the validation set, so nobody can win by stopping early at a worse model.¹ The rules pin the rest: in the Closed division a submission must use the same preprocessing, model, training method, and quality target as MLCommons' reference implementation.²

Inference is a separate suite with its own metrics. Instead of one number it defines scenarios that stand in for real request patterns. In the Offline scenario every sample is available at once and the score is throughput in samples per second; in the Server scenario requests arrive at random times and the score is the queries per second the system holds under a latency bound.³ A submission reports one of these per benchmark, and for a language model that throughput is counted in tokens per second.⁴

So an MLPerf number is never just fast. It is a specific quantity, a time or a throughput under a named scenario, for one system on one fixed task under one set of rules.

What a division and a category fix

A number for one system earns its keep only next to a number for another, and two MLPerf numbers are comparable only when they were made under matching rules. Two labels do most of that work. The division names the rulebook the run followed. Closed pins the reference model and method, so the only thing left to differ is the systems. Open lets the submitter change the model or the training method, which turns the run into a demonstration rather than a like-for-like race.² The availability category says how real the system is. Available means you can rent or buy it now. Preview means the submitter has committed to ship it within a set window. RDI, for research, development, or internal, means a lab machine that may never ship. These categories apply to Closed, Network, and Open runs alike.⁵

MLCommons does not leave the comparison to trust. Its Results Messaging Guidelines require that any published comparison name every dimension of the box, chip count included, where chip count is the count of the compute devices doing the most of the work.⁶

The document even works an example for you: an eight-chip result in the RDI category of the Closed division may be set against a sixteen-chip Available on-premise Closed result only if both descriptions sit on the page.⁶ A comparison that crosses those labels, or drops them, is not one the rules allow.

256 chips against 512

To see why chip count belongs inside the number, take one round. MLPerf Training v5.0, published in June 2025, carried 201 results from 20 organizations.⁷ In it, one submitter ran the same benchmark, Llama 3.1 405B pretraining, on the same accelerator at two sizes. On 256 GB200 GPUs the run reached the quality target in 205.6 minutes; on 512 of the same GPUs it took 106.5 minutes.⁸ Everything else held fixed: same benchmark, same Closed division, same Available on-premise category, same chip. Only the count moved. These are the wall-clock times from each run's own timing log, one run apiece, so read them to the minute rather than as the single figure MLCommons selects across a benchmark's several runs.⁸

Now do the division a headline would skip. 205.6 divided by 106.5 is 1.93. Doubling the chips made the run 1.93 times as fast.⁸ That is about what you would want from twice the hardware, and it is a fact about scale, not about a better chip. Put the 106.5-minute figure on a slide without the chip count and it reads as a verdict on the silicon; it is really a verdict on how many of that silicon someone bought.

That comparable pair is also the ceiling on what the number supports. Under Closed rules it is a clean system-against-system time on a fixed task. It is not a claim that the model is better, because the model is fixed. It is not a promise about your own workload, which is not the benchmark. And it says nothing across a boundary. A 512-chip time cannot stand next to a 256-chip time as though the systems were peers, and that is the exact difference the guidelines make a submitter disclose.⁶

Where the disclosure gets dropped

None of this is hidden inside MLPerf. The box is printed in every row, and the guidelines demand it be repeated in every comparison. The failure happens later, when a marketing page or a news write-up keeps the ratio and drops the disclosures around it.

The clearest documented case sits in Google's 2023 paper on its TPU v4. Reading the published MLPerf Training 2.0 leaderboard, the authors note that the largest A100 entry ran on 4,216 chips while the largest Graphcore IPU entry ran on 256, a roughly sixteenfold gap in scale that the raw ranking simply lists side by side.⁹ Their own summary makes the trap concrete. A four-times-larger, 4,096-chip system came out nearly ten times faster. Compare systems of similar size and the TPU v4 lead over the IPU is about 4.3 times on one benchmark and 4.5 on another, well short of the gap the raw table implies.⁹ MLCommons, the authors add, lets a vendor pick the size of system it reports and does not require an equal-size comparison.⁹ That reading is a competitor's, since Google submits too, but it lines up with the rules and with the v5.0 scale pair above.

The honest version of the same move is easy to tell apart. In the v5.0 round, NVIDIA reported its Blackwell system training Llama 3.1 405B 2.2 times faster than the previous Hopper generation, 512 GPUs against 512 GPUs, and up to 2.6 times the performance per GPU across the suite.¹⁰ Chip count, benchmark, division, and category are all held fixed and only the hardware generation changes, which is what the rules allow. The number and the disclosure travel together.

The pattern is still live. Covering MLPerf Inference v6.0 in 2026, one trade report noted that NVIDIA's throughput record came from the largest configuration ever submitted: 288 GPUs wired into a single system, against the far smaller machines AMD and Intel entered. The results, the report wrote, are only partially comparable, because each vendor frames the configuration that shows it best.¹¹ The record is real. It is a record for 288 GPUs, and a reader who wants to know about eight has to go find that line.