LoRA matched full fine-tuning by training 4.7M of GPT-3's 175 billion weights

A frozen model with a small pair of matrices bolted on

Fine-tuning means continuing to train a finished model on new data, nudging its weights by gradient descent until it does a new job. For a small model that is cheap. For GPT-3, with its 175 billion parameters, it is not. Full fine-tuning updates every one of those weights, needs enough memory to hold a gradient for all of them, and leaves you with a fresh 350-gigabyte copy of the model for each task you train. Do that for ten customers and you are storing ten GPT-3s.

LoRA, short for Low-Rank Adaptation, removes almost all of that cost with one change. It freezes the pretrained weights, so none of the original 175 billion numbers move during training. Beside a weight matrix it wants to adapt, it adds a detour: a pair of much smaller matrices, call them A and B, and trains only that pair.¹ A takes the layer's input and squeezes it down to a handful of numbers. B expands that handful back to the layer's full width and adds the result to what the frozen weights already produce. If A projects down to eight numbers, then A and B together hold a few thousand weights where the frozen matrix holds millions.

In the GPT-3 experiments the authors did not even adapt every matrix. They inserted the low-rank pairs only into the query and value projections inside each attention layer, the part that decides what each position reads from the rest of the sequence, and set the squeeze as narrow as four numbers.¹

Training touched about 4.7 million weights instead of 175 billion. The memory needed to train fell from 1.2 terabytes to 350 gigabytes, a threefold cut. And the thing you save at the end, the trained adapter, is 35 megabytes rather than a 350-gigabyte model, about ten thousand times smaller.¹ The ten customers now cost you one GPT-3 and ten small files.

The change in the weights was low-rank all along

Why should a few million numbers be enough to teach a 175-billion-weight model a new task? The paper's answer is a guess about what fine-tuning actually changes. When you fine-tune, every weight is free to move, but the authors proposed that the total change is far simpler than that freedom allows. It has, in their words, a low "intrinsic rank."¹

Rank measures how much independent variety a grid of numbers holds. A matrix of weight changes could in principle push the model in as many independent directions as it has rows. The claim is that it does not: the update a task needs can be rebuilt from a handful of directions, so writing that update as a narrow A-times-B product throws almost nothing away. That is why B can expand a squeeze of eight numbers rather than eight thousand. The authors tested it directly. Sweeping the squeeze from a rank of 64 down to 1, accuracy on their tasks barely moved, and the paper reports that a rank as small as one suffices for adapting those query and value matrices.¹

The idea did not come from nowhere. A year earlier, Aghajanyan and colleagues had shown something adjacent and striking: they tuned RoBERTa to 90 percent of its full performance on one task by optimizing only 200 numbers, projected at random into the full weight space.² That measured how few dimensions fine-tuning needs inside some random subspace. LoRA's contribution was the sharper, more useful bet: that the weight change itself is low-rank, and so can be written as an explicit small product you train and then fold away.

The trained pair folds back into the frozen weights

A detour beside every attention layer sounds like it should make the model slower to run. Earlier methods that inserted extra trainable layers, called adapters, did exactly that: they added computation the model had to perform on every token, at every use, for as long as the adapter was attached.¹ LoRA does not, and the reason is arithmetic. The trained pair A and B multiply out to a matrix of exactly the same shape as the frozen weight matrix beside them. Once training is done you add that product into the frozen weights a single time and discard the pair.¹ What remains is an ordinary model of the original size and speed, with the new behavior baked into its weights. No detour is left to run.

This is the property that sets LoRA apart from the adapters before it, and it is why a base model and its adapter can be shipped, swapped, and merged as casually as they are now. The 35-megabyte file is a patch: apply it to the base weights and the adaptation costs nothing at the moment of use.

Parity, inside the paper's own benchmarks

The headline claim is that this cheap method matches the expensive one. On the tasks the paper measured, it does, and here and there it wins.

Read down the trainable-parameter column and the gap is stark: LoRA trains between a thousandth and a ten-thousandth of the weights and lands on the same scores.¹ Then read what the tasks are. GLUE is a suite of sentence classification problems. E2E and WikiSQL turn short structured inputs into short outputs. MNLI asks whether one sentence follows from another. These are real benchmarks, and they are also narrow: constrained inputs, constrained outputs, no long body of new knowledge the model has to take on. Matching full fine-tuning is a result on tasks like these, not a law of adaptation.

A 2024 study, "LoRA Learns Less and Forgets Less," pushed on exactly that boundary. Training Llama-2-7B on programming and mathematics, it found that LoRA substantially underperforms full fine-tuning.³ The gap was sharpest in the hardest case: continued pretraining on tens of billions of new tokens, where the model must absorb a large new domain rather than adjust to a task. The same study offers a mechanism that fits the original hypothesis instead of breaking it. Full fine-tuning, it found, learns weight changes of a rank ten to a hundred times higher than the ranks LoRA typically uses.³ When the update a task needs is genuinely low-rank, LoRA loses nothing. When the task needs a richer change than a narrow squeeze can carry, LoRA has no way to express it.

That constraint has a compensating side. Because LoRA moves the model less, it also disturbs it less: the same study found that LoRA better preserved the base model's other abilities and produced more varied output.³ Training fewer parameters buys the limitation and the safety in one stroke.

Now almost every fine-tune is a LoRA

Five years on, the method is the default. Of the model cards on the Hugging Face Hub that name a single parameter-efficient technique, 98.4 percent name LoRA, 20 509 out of 20 834; among image-generation checkpoints the share is 95 percent.⁴ When someone says they fine-tuned an open model, they almost certainly mean this.

A 2023 successor, QLoRA, pushed the cost down again by storing the frozen base model in four-bit numbers while training the same low-rank pair on top. It fit the fine-tuning of a 65-billion-parameter model onto a single 48-gigabyte GPU, the kind one person can rent by the hour.⁵ Its authors reported that the result reached 99.3 percent of ChatGPT's level, though on a benchmark scored by GPT-4's preferences rather than by task accuracy, a softer measure than the tables above.⁵

What travels with all this reach is the parity claim, stripped of its scope. "Matches full fine-tuning" gets repeated as though it held everywhere, when the paper only ever showed it on constrained language tasks¹ and a later study found it trailing on code and math.³ The standard open-source toolkit for the method, Hugging Face's PEFT library, demonstrates LoRA first among its options.⁶ The stand-in has become so routine that the scope drops off the claim. For most fine-tunes, on tasks as constrained as the paper's, that is harmless. For teaching a model a large new domain like code or math, it lands exactly where the expensive method still does something the cheap one cannot.