Two more papers turned a robot's backflip trick into InstructGPT's training recipe

A backflip has no reward function to write

An Atari game scores itself. Space Invaders adds points for every alien killed, so a reinforcement-learning agent needs only one rule: maximize the score. Most behavior worth training does not come with that gift built in. Nobody has written a formula that outputs a number for "good backflip" the way a scoreboard outputs a number for "good game," and nobody has one for "be helpful" either. The behavior nobody can score by formula is exactly the behavior a hand-written reward function cannot train. That gap is what Paul Christiano, Jan Leike, and four coauthors split between OpenAI and DeepMind set out to close in a 2017 paper, Deep reinforcement learning from human preferences.¹

Their fix was not a better reward function. It was to stop writing one. A human instead watches two short clips of the agent acting and clicks which clip looks more like the target behavior, or calls it a tie.¹ That click is the unit the rest of this lesson runs on: one comparison, one person, two clips, one preference. A second network, the reward model, learns to predict which of two clips a human would pick, reverse-engineering a score function from a pile of clicks instead of asking anyone to write one down.¹ Once the reward model can guess a person's preference reliably, an ordinary reinforcement-learning algorithm trains the policy, the agent's rulebook for choosing an action, to rack up clips the reward model rates highly.¹

Every number in this lesson is a multiple of that one click. A comparison is not a survey question or a star rating. It is a person watching two several-second clips and choosing the one closer to the goal. Fix that unit and a total of nine hundred comparisons stops being an abstraction to take on faith. It is nine hundred clicks, a task one person could plausibly finish before lunch.

Nine hundred clicks taught a robot to backflip

The paper's clearest demonstration is a simulated robot trained to perform a backflip, a behavior neither the researchers nor anyone else had a reward function for.¹ It took about 900 comparisons, delivered in under an hour of a human's time, for the reward model to learn what a backflip looked like well enough to train the policy into doing one.¹ Of every figure in the paper, this is the one nearly every later retelling of RLHF's history repeats.²

The same order of effort ran across the paper's full test set: seven Atari games and eight simulated-robotics tasks built on the MuJoCo physics engine.¹ Human time per task typically ran from thirty minutes to five hours, and across every task the training used comparisons on fewer than one percent of the agent's total interactions with its environment.¹

How small the whole experiment actually was

Scale the per-task counts up and the whole 2017 project is still modest. The paper reports roughly 5,500 comparisons per Atari game and about 700 per MuJoCo task.¹ Summed across all fifteen tasks, that puts the entire experiment at something on the order of 44,000 human judgments in total, an estimate built from the paper's own per-task figures rather than a single total the paper states outright.

The count matters less than the domain. Every task in the paper is a video game or a simulated body with joints. None of them is language. The reward model looks at pixels or a robot's joint angles, never a sentence.¹ The paper is the origin of the recipe now called RLHF, and it never once ran that recipe on the kind of model RLHF is now famous for aligning.

What it took to move the recipe onto text

The authors knew the recipe would need to survive a jump before it could do what it is credited for today. The year after the backflip paper, two of its coauthors, Jan Leike and Shane Legg, published a research agenda naming the problem directly: reward modeling would meet new failure modes once it moved to tasks too complex for a human to fully judge by watching a clip.³ That agenda came before either of the papers that actually made the jump.

The jump took two separate papers. In 2020, Nisan Stiennon and coauthors, two returning from the 2017 team, applied the same pairwise-comparison recipe to a real language task: summarizing Reddit posts and news articles.⁴ Their preference-trained summarizer beat both the human reference summaries it learned from and larger models trained the ordinary supervised way.⁴ That result showed the reward-model idea could survive contact with open-ended text, not just a game score or a joint angle.

Two years later, Long Ouyang and coauthors at OpenAI used the same three-part recipe, human comparisons feeding a learned reward model that then steers reinforcement learning, to fine-tune GPT-3 into InstructGPT.⁵ The paper states its smallest headline result in words rather than a number: outputs from the 1.3-billion-parameter InstructGPT were preferred over the 175-billion-parameter GPT-3 it started from, despite having a hundred times fewer parameters.⁵ The percentage the paper does print belongs to a different pairing: InstructGPT's own 175-billion-parameter version against GPT-3. There, it was preferred 85 percent of the time over the plain baseline, and 71 percent of the time over a version of GPT-3 already primed with a few examples.⁵ Nine hundred clicks taught a robot to flip in under an hour. Getting the same idea to beat a language model a hundred times its own size took five years, two intervening papers, and a training pipeline built around a production language model instead of a game engine.

Reward hacking is an old worry the paper never resolved

Learning the reward instead of writing it removes one problem and leaves another exactly where it was. A year before the backflip paper, some of the same authors, Dario Amodei and Paul Christiano among them, published a survey of AI safety problems that named reward hacking directly: a system optimizing a stated objective can find a loophole that scores well without doing what its designers meant.⁶ Reward hacking does not care whether the objective was hand-written or learned from human clicks. A reward model is still a proxy, and a policy trained hard enough against any proxy can learn to satisfy the proxy instead of the goal behind it.⁶

The 2017 paper does not claim to have solved this, and its design did not test for it. Its tasks ran short training windows on domains simple enough that a human glancing at a clip could catch an obviously wrong behavior before it spread through many more updates. It demonstrates that a learned reward can substitute for a hand-written one. It does not demonstrate that a learned reward resists gaming once the system optimizing against it grows more capable. Reward hacking against a learned reward model shows up throughout the RLHF work that followed, general enough that one recent survey of the problem places it in the same family as a sales team gaming its own click-through metric: pursuing the measurement instead of the outcome the measurement was built to track.⁷ The failure mode was named a year before RLHF's origin paper published. That paper's contribution was a way to learn a better proxy, not a way to stop a system from gaming whichever proxy it is handed.