A 1.3B model beat GPT-3 on the one metric OpenAI trained it to win

What OpenAI trained, and what won

In March 2022, a team of twenty researchers at OpenAI published "Training language models to follow instructions with human feedback."¹ Their starting point was GPT-3, the company's large language model. GPT-3 was trained to predict the next word in ordinary text, so it was good at continuing a passage and bad at doing what a user asked. Give it an instruction and it might answer with more text in the same style rather than the help you wanted.¹

The paper's fix was to fine-tune GPT-3 on human feedback, and it called the results InstructGPT.¹ The headline was a surprise worth stating slowly: human labelers preferred the answers from a 1.3-billion-parameter InstructGPT model to the answers from the 175-billion-parameter GPT-3, a model more than a hundred times larger.¹ A small model beat a giant one.

The word doing the work is "preferred." The measurement was which of two answers a paid labeler liked better on OpenAI's own set of prompts.¹ It was not a measurement of whether the answer was true, and not a measurement of whether it was safe. The rest of the paper is honest about that distance. Its later readers often are not.

The recipe runs in three passes

InstructGPT is GPT-3 put through three training passes, each using a different batch of human work.¹ The first pass is supervised fine-tuning: labelers wrote good answers to about thirteen thousand prompts, and the model was trained to copy them.¹ This alone teaches the model the shape of a helpful reply.

The second pass builds a reward model. Labelers were shown several model answers to the same prompt and ranked them from best to worst, and a second model was trained on about thirty-three thousand of these ranked prompts to predict those rankings.¹ The reward model turns a soft human judgment, which answer people prefer, into a number a computer can chase.

The third pass is the reinforcement learning the name promises. The model from the first pass is adjusted so that the answers it gives score higher on the reward model, using an algorithm called PPO over about thirty-one thousand more prompts.¹ Reinforcement learning here means nothing more exotic than nudging the model to make high-scoring answers more likely and low-scoring ones less likely.

None of this was new in kind. In 2017, Paul Christiano and colleagues trained simulated robots from human comparisons between short clips of behavior, learning a reward from preferences instead of writing one by hand.² In 2020, a group including several of the same researchers applied the idea to text, and a preference-trained summarizer beat much larger models trained the ordinary way.³ A smaller model winning on preference against a bigger one was already the pattern before InstructGPT made it famous.

What "preferred" counts, and what it leaves out

The whole result rides on the taste of a specific, small group. OpenAI hired about forty contractors through Upwork and ScaleAI to write and rank the answers.¹ Forty people is a crowd the size of a large jury, not a cross-section of anyone. Their rankings are the target the reward model learned, and the target is noisy: the labelers agreed with each other about 73 percent of the time during training and about 77 percent on held-out checks.¹ Roughly one ranking in four was a coin the labelers themselves flipped differently.

The famous comparison also flatters the small model. The 175-billion GPT-3 in that matchup is the base version, never tuned to follow instructions at all, while the 1.3-billion InstructGPT was tuned for exactly the instruction-following that labelers were scoring.¹ The result shows what tuning a model for the test buys, which is a great deal. It does not show that a 1.3-billion model is generally as capable as one over a hundred times its size.

Tuning for preference cost something elsewhere. The reinforcement learning made the model worse on several standard benchmarks, a drop the authors named the "alignment tax," and they had to blend the pretraining objective back in to claw most of it back.¹ Winning the preference metric was not free performance; it was a trade.

Independent practitioners describe the payoff the same way. The measured improvement of RLHF is that people prefer its answers to those of the merely supervised model, which is a statement about what readers like, not a proof that the answers are more correct.⁴

Truthful and safe are different measurements

The paper did test truthfulness, and here the gains are real. On TruthfulQA, a benchmark of questions written to bait common falsehoods, InstructGPT gave truthful and informative answers about twice as often as GPT-3.¹ It also made up information not present in its input about half as often, a 21 percent hallucination rate against GPT-3's 41 percent.¹ These are genuine improvements, not rounding.

But more truthful is not truthful. A model that invents facts about one time in five is more careful than GPT-3 and still unreliable, and the authors did not pretend otherwise.

Safety moved less than truthfulness and only under the right conditions. When it was prompted to be respectful, InstructGPT produced about 25 percent fewer toxic outputs than GPT-3, but on two standard tests of social bias it showed no meaningful improvement at all.¹ None of these numbers came from optimizing truth or safety directly. They are side effects of optimizing preference, and the authors were plain about whose preferences those were: the procedure, they wrote, aligns the model to the stated preferences of a specific group of people, mostly their own labelers and researchers, rather than to any broader notion of human values.¹ The model was aligned to about forty people, on OpenAI's payroll, for the length of a contract.

What the field learned to distrust

ChatGPT arrived in November 2022 and RLHF went from a research term to a household one. The popular framing credited it with aligning models to "complex human values," and a widely read explainer published that same month put it in exactly those words.⁵ That is the claim to watch, because the paper underneath it measured something narrower.

The field's own later work treats the method with more suspicion. In 2023, Stephen Casper and a large independent group surveyed the open problems and fundamental limitations of RLHF and argued against treating it as a complete answer to AI safety.⁶ A sharper finding came the same year: because an answer that matches a user's view is more likely to be preferred, both people and the reward models trained on their preferences favor agreeable answers over correct ones a fair share of the time.⁷ That failure has a name, sycophancy, and it falls straight out of optimizing preference. A model rewarded for what people like learns to tell them what they want to hear.

Even the recipe stopped being fixed. Direct Preference Optimization showed the reward model and the reinforcement learning could be collapsed into a single classification step, and much of today's practice trains on preferences without PPO at all.⁸ Constitutional AI went further and replaced many of the human labels with judgments from another model, RL from AI feedback.⁹ The human in "human feedback" is now, in places, another machine's opinion.