PaLM's training ran at 46 percent efficiency, or 58, depending on which FLOPs you count

A fraction with a measured top and a spec-sheet floor

Model FLOPs utilization, MFU, is a fraction. The top is the arithmetic a model actually had to do in a second of training. The bottom is the most arithmetic the chips could have done in that same second. A FLOP is one floating-point operation, a single multiply or add on the decimal numbers a neural network is built from. Peak FLOPs is the largest number of those operations a chip's spec sheet claims it can run per second, a ceiling the hardware reaches only under laboratory conditions.

When Google trained PaLM, its 540-billion-parameter language model, on 6,144 of Google's TPU v4 chips, it reported that the run reached 46.2 percent MFU.¹ Read plainly, the model's necessary math ran at a little under half the rate the silicon could in principle sustain. The same paper reported a second number for the identical run, 57.8 percent, under a different name, hardware FLOPs utilization.¹ Both are honest, and the distance between them is the first clue that "efficiency" here depends entirely on what you agree to count.

Six FLOPs per parameter, per token

Start with the top of the fraction, because it is the honest part. To fill it you have to count the FLOPs a model must do, and there is a standard way. Take a dense model with N parameters. Running one token forward through the network costs about 2N FLOPs, because each parameter multiplies an activation and the products are summed.² Training also runs the token backward to compute gradients, which costs roughly twice the forward pass, so the round trip lands at about 6N FLOPs for every token.² Kaplan's scaling-laws paper wrote it as C \approx 6N per training token. DeepMind's Chinchilla work used the same 6ND for a whole run and checked the shortcut against a detailed operation-by-operation count, finding it accurate to within about ten percent across model sizes.³ It is the same six-per-parameter figure a companion lesson used to rebuild a run's total training compute.

Now the worked number. PaLM has N of 540 billion, so each token costs about 6N, which is 3.24 trillion FLOPs. The run pushed 238,300 tokens every second.¹ Multiply the two and the model's own math ran at roughly 7.7 times ten to the seventeenth FLOPs per second. For the denominator, each TPU v4 chip peaks at 275 trillion bf16 FLOPs per second, and there were 6,144 of them, for about 1.69 times ten to the eighteenth FLOPs per second of peak.⁴ Divide the first by the second and you get 0.46. The bare 6N estimate reproduces the reported 46.2 percent almost exactly; PaLM's own count adds a small attention term on top of 6N, which is what carries it the last few tenths of a point.¹

The denominator the chipmaker defines

The numerator is grounded. 6N is arithmetic, checked against detailed counts.³ The denominator is where MFU turns soft, because "peak FLOPs" is not a single number. A chip has a different peak for every numeric precision it supports, and the headline a vendor prints often assumes a hardware trick called structured sparsity, which skips half the multiplications when half the weights are forced to zero. The precision choice matters because training in a coarser number format lets each core do more operations per second, so the same chip is faster in fp8 than in bf16, and faster in bf16 than in the older fp32.

Take NVIDIA's H100, the chip most current models train on. Its spec sheet lists 989 teraFLOPS at TF32, 1,979 at bf16, and 3,958 at fp8.⁵ Every one of those numbers is marked "with sparsity," and the fine print states that the dense figure, without the trick, is exactly half.⁵ So a single H100 has a bf16 dense peak near 990 teraFLOPS and an fp8-with-sparsity headline of 3,958, four times larger, with two more peaks in between.

Here is the trap that follows. A lab trains in bf16 and honestly divides its achieved FLOPs by the dense bf16 peak near 990, reporting, say, 47 percent MFU. A reader who grabs the datasheet's biggest number, 3,958, and treats it as "the H100's peak" divides the same achieved FLOPs by four times as much and gets about 12 percent. Same chip, same run, the very same arithmetic on top, and the efficiency reads 47 percent or 12 percent purely by which line of the spec sheet went underneath.⁵ Two MFU figures are comparable only when both name the precision and say whether sparsity was assumed. Absent that, the number is a fraction with an undisclosed bottom, and it can be inflated or deflated to taste without anyone touching a transistor.

What a utilization number cannot see

Used honestly, the fraction is worth having, because it bounds headroom. At 46 percent, PaLM's run could at most a bit more than double its throughput on the same chips before it hit the peak, and no rewrite of the code can push past that ceiling.¹ That makes MFU a concrete target for the engineers whose job is to close the gap, and a fair yardstick between two runs on the same chips at the same precision. The Megatron work that carried a 530-billion-parameter model to 56 percent MFU on 2,240 A100 GPUs was measuring exactly that kind of progress.⁶ The ceiling is never actually touched: open-source training of GPT-NeoX reached about 150 of an A100's roughly 312 teraFLOPS, near half of peak, which is ordinary for a well-tuned run.⁷

What the fraction cannot see is whether the FLOPs were worth doing. MFU counts the model's arithmetic against the clock and says nothing about whether that arithmetic trained a good model, converged at all, or ran on well-chosen data. A diverging run and a healthy one can post the same MFU. The measure is blind to a second kind of waste as well. PaLM's hardware utilization of 57.8 percent runs higher than its model utilization of 46.2 percent because the run recomputed some activations to save memory, real work the chips performed that the model did not strictly require.¹ MFU leaves that recompute out by design, which is what makes it the honest figure for the model's own efficiency rather than the chips' busyness.

Comparisons across setups fail for the same reason the H100's peaks spread out. Megatron's 56 percent was divided by an A100's 312 teraFLOPS and PaLM's 46 percent by a TPU v4's 275, while a run on an H100 divides by something near 990.⁶⁴⁵ A lower MFU on the faster chip can reflect only its bigger denominator. Epoch AI, surveying real training runs, tells its readers to assume about 30 percent for a large language model and treats anything from a third to three-quarters of peak as unremarkable.⁸