Quantization rounds a model's weights to a coarser grid, and a few oversized values decide what that costs

The cheaper copy answers worse

Run the small, on-device build of a language model beside the full one and the small build often gives a weaker answer, though both are sold under the same name. What changed is not the network's design and not its training. It is how the network's numbers are stored.

Those numbers are its weights, the values training left behind. By default each one is held in a 16-bit floating-point format: sixteen bits of storage per weight. A modern model has billions of them, and because there are so many, the weights are almost the whole of the model's memory. The arithmetic is blunt. GPT-3 has 175 billion weights. At sixteen bits each, they occupy 326 GB, more than any single accelerator holds, enough to require five 80 GB A100 GPUs just to load the model.¹ An independent measurement puts the same footprint at 350 GB.² That pressure, to make the model fit in less, is what the whole practice answers, and the name for the practice is quantization.

Fewer bits is a coarser grid

Quantization stores each weight using fewer bits, an 8-bit or a 4-bit integer in place of the 16-bit float. Fewer bits means fewer distinct values to land on. Four bits offer sixteen levels in all. Picture a ruler marked only at every quarter inch: any length you measure gets read off as the nearest mark, and a value that falls between two marks is reported as whichever is closer. Quantization does that to every weight, snapping each to the nearest value the coarser grid allows. The error introduced in any one weight is small. There are billions of weights, so whether those small errors stay small is the entire question.

The payoff is immediate. A 7-billion-weight model held at sixteen bits is 13.0 GB on disk; stored at four bits it is 3.80 GB, about a third the size, small enough to run on a laptop or a consumer graphics card.³ The weights carry this saving because they dominate the memory in the first place: a model has billions of weights against only millions of the smaller quantities beside them, so the effort goes where the bytes are.⁴ This is a different move from distillation, which trains a smaller network to copy a bigger one. Quantization keeps the same network and rewrites each weight in fewer bits. The first thing it buys is room. Speed follows only where there are fewer bytes to move for each weight, which is why the weight-only 4-bit tools run faster while the memory-only 8-bit schemes need not.¹

Down to four bits, almost free

The tool most on-device users actually run publishes exactly what each bit-width costs. It measures that cost as perplexity, the model's surprise at held-out text, where lower is better and the 16-bit model sets the baseline.³ Eight-bit storage is free to two decimal places. Six-bit stays within a tenth of a percent. Four-bit, the recommended default, costs under one percent while cutting the file by more than three-quarters. The cost stays flat across that range and only begins to climb once the grid drops below four bits.

The flat top is not a quirk of one tool. A study spanning more than 35,000 runs, across bit-widths from three to eight and model sizes from 19 million to 176 billion parameters, found four-bit precision almost universally the best trade of total storage against accuracy.⁵ Above four bits you spend storage you do not need. So the tidy version of the story is real: down to about four bits, a good scheme costs almost nothing. Both of the words holding it up, good and about four, are doing more work than they look.

One big value ruins the grid for the rest

Start with good. The cheap results above come from schemes that work to place the grid well. The naive alternative, round-to-nearest, simply snaps every weight to its closest level under a single scale for the whole tensor, and it is not cheap where the model is large. On OPT-175B, a careful 4-bit scheme scores 8.37 in perplexity against the 16-bit model's 8.34. Naive 4-bit rounding on the same weights scores 10.54. Naive 3-bit rounding sends perplexity to roughly 73,000, a model that has stopped producing sense.¹ The gap widens with scale rather than closing: naive 8-bit rounding scores worse at 13 billion parameters than at 6.7 billion, while an outlier-aware method holds the full-precision baseline at both.⁶

The reason is a small set of unusually large numbers. Past roughly 6.7 billion parameters, a handful of the model's internal feature dimensions begin carrying values far larger than the rest, up to twenty times larger. About 150,000 of these outliers appear in a single 2,048-token pass, concentrated in fewer than seven dimensions. They arrive suddenly, in a phase shift between 6 and 6.7 billion parameters, after which every layer is affected.⁶

Those few dimensions carry the model. Zero them out, though they are only about a tenth of a percent of the features, and perplexity rises by 600 to 1,000 percent. Zero the same count of ordinary features chosen at random, and perplexity moves about a tenth of a percent.⁶ Under a single scale for the whole tensor, one such outlier stretches the range the grid has to cover, and every ordinary weight is then rounded on a grid spread too thin to hold it. That is why naive rounding collapses precisely on the largest models: their outliers break the quantization of everything around them.¹

The good schemes are the ones that refuse to let that happen. One keeps the outlier dimensions in 16 bits and quantizes the other 99.9 percent to 8 bits, which recovers full-precision quality up to 175 billion parameters and still cuts memory by about half.⁶ Another protects the roughly one percent of weight channels the largest activations flow through, picking them by those activations rather than by the weights themselves. It also scales the weights in small groups, so no single value sets the range for a whole tensor. Keeping that one percent of an OPT-6.7B model precise pulls its naive 3-bit perplexity from 23.54 back to 11.39, close to the 10.86 of the full model.⁷ The same large-value problem also appears in the model's activations, where it can be shifted offline into the weights by an exact, offsetting rescale of each channel, so both sides quantize cleanly to 8 bits.²

Below four bits, the floor gives way

Protecting the outliers buys the flat top down to four bits. It does not buy the range below. Even a careful scheme loses real ground once the grid gets coarser than four bits. Llama-2-7B rises from 5.47 in perplexity to 5.60 at four bits, a tenth of a point, then to 6.24 at three bits, about a seventh worse.⁷ The on-device tool's two-bit setting costs nearly 15 percent.³

Perplexity can also understate how bad three-bit gets. On an image captioning task, a nine-billion-parameter model scored on how well its captions match the reference lost 16.9 points at naive three-bit, against 1.17 for a good four-bit scheme.⁷ The number a person would actually notice fell much further than the surprise score alone would warn. Rounding is cheap for good schemes down to about four bits, and it is expensive when the scheme is naive, when the model is large, or when the grid drops below four bits, three conditions the tidy version leaves out.

Two questions here are open even for the people who build these models. The first is why the outliers arise at all. The sudden shift near 6.7 billion parameters is documented, and an earlier finding ties some outliers to dimensions that push attention onto a sentence's separator token, but no account settles why the model grows them.⁶⁸ The second is where the floor sits. Four bits is the best measured trade of accuracy for storage, and pushing lower loses accuracy faster than it saves memory, but how far a model can safely be pushed below four is not settled.⁵