DPO turns preference tuning into a single classification loss
Its simplicity won open models over from the RLHF pipeline, though the evidence that it matches PPO was small-scale and later work still puts PPO ahead on the hardest tasks.
Why this matters
Almost every open chat model you can download was tuned with DPO or a close variant, and the reason usually given is a single line: it is as good as the older RLHF method and far simpler. This lesson is about what that line hides. You will see what DPO actually does, how much evidence the original paper offered for the comparison, and where later work still finds PPO ahead. The aim is that "trained with DPO" stops being a slogan you take on faith and becomes a claim you can size up.
- 01
- A 1.3B model beat GPT-3 on the one metric OpenAI trained it to win: the RLHF pipeline DPO simplifies, in three stages.
- 02
- Two more papers turned a robot's backflip trick into InstructGPT's training recipe: where learning a reward from human comparisons began.
What "trained with DPO" actually names
Anyone reading about how chat models are built has met the phrase "trained with DPO," usually next to a claim that it works about as well as RLHF, the reinforcement-learning method this course covered earlier, and is far simpler to run. The 2023 paper that introduced it, by Rafael Rafailov and five coauthors at Stanford, makes that claim directly: its method matches or improves on reinforcement-learning tuning "while being substantially simpler to implement and train."1 The slogan leaves out the part worth understanding. Simpler how, and simpler than what? And if a method this much lighter really matched the one it replaced, what was the heavier method doing that turned out to be unnecessary?
One loss instead of a reward model and a loop
Start with what DPO removes. The RLHF recipe from OpenAI's 2022 InstructGPT work runs in three stages. First it fine-tunes a model on example answers. Then it trains a second model to score answers the way human raters did. Finally it runs a reinforcement-learning loop that pushes the first model toward high-scoring answers, with a tether holding it near where it started.2 The scoring model and the loop are the expensive stages. Each needs its own training and tuning, and the loop keeps sampling fresh answers from the model as it learns.
All of it rests on one cheap ingredient: people comparing two answers and saying which is better. Learning a reward from such comparisons, rather than from a rule someone wrote by hand, is an idea this course traced to a 2017 paper that trained simulated robots from human choices.3 RLHF gathers those comparisons, distills them into the scoring model, and only then optimizes against it.
DPO's finding is that the scoring model and the loop can be skipped without giving up what they computed.
For any way of scoring answers, there is one best-possible answer-model, and its shape is fixed by that scoring rule. Read that backwards and the answer-model already carries the score inside it, so the separate scorer is redundant. DPO uses this to train the model directly on the ranked pairs: for each pair, raise the model's probability of the preferred answer and lower the rejected one, both measured against the untuned model it began as. No scoring model, no reinforcement-learning loop, no sampling during training.1
What is left is a single classification loss, the same kind of objective used to sort spam from not-spam. One loss in place of a scoring model and a reinforcement-learning loop can read as too small a thing to work, and that reaction is worth keeping. The two stages it drops were not doing nothing. Whether skipping them costs anything is a question the paper has to settle with evidence.
The three tasks behind the parity claim
The paper tests DPO against PPO, the reinforcement-learning algorithm RLHF actually uses, on three tasks. All three run on small models by current standards: GPT-2-large, at 774 million parameters, for a sentiment task; a 6-billion-parameter model for summarization; and Pythia-2.8B for dialogue.1
| Task | Model | What the result compared |
|---|---|---|
| Sentiment | GPT-2-large | DPO vs PPO, with the reward computed by a sentiment classifier rather than by people. DPO reached higher sentiment control at the same distance from the start model. |
| Summarization | 6B | DPO vs PPO, GPT-4 judging summaries against human-written ones. DPO won about 61% of the time to PPO's best 57%; human raters preferred DPO 58%. |
| Dialogue | Pythia-2.8B | DPO vs Best-of-128 sampling. No working PPO baseline existed, so Best-of-128 stood in for it. DPO topped the dataset's own chosen answers. |
On summarization, a judge model was shown two summaries and asked which it preferred against the human-written reference. DPO won that comparison about 61 percent of the time at low sampling temperature, against PPO's 57 percent at its own best setting, and human raters shown the same pairs preferred DPO's summaries 58 percent of the time.1 DPO also held its lead as the sampling temperature rose, where PPO's win rate fell away.
Two things about that evidence are easy to miss. The sentiment result, the one place DPO clearly beat PPO, used a reward computed by a sentiment classifier rather than by people, so it measures control of a known target, not agreement with human taste. And on dialogue, the paper's hardest task, there is no real PPO number at all. The authors report they could not find a setting at which their PPO model beat the untuned base model, so they substituted Best-of-128 sampling as a rough stand-in.1 On the task closest to how these models are actually used, the head-to-head with PPO is a substitution, not a measurement.
A win rate a few points apart, on models this size, judged mostly by another model, is a real result and a narrow one. It supports "about as good as PPO, and far simpler." It does not, on its own, support "DPO replaced RLHF."
Open models made it the default anyway
Whatever the paper proved, DPO won the thing that decides which method people use. It became the default recipe for tuning open models. The clearest single case is Zephyr, a 2023 model from Hugging Face built by running DPO on a 7-billion-parameter base, using preference data scored by GPT-4, with no human annotation and no sampling during tuning. Zephyr-7B scored 7.34 on MT-Bench, a chat benchmark, beating the 6.86 of Llama-2-Chat-70B, a model ten times its size tuned with the full RLHF pipeline.4 DPO and its variants have since become a standard post-training step across open models, prized for exactly the simplicity the paper claimed.5
Adoption is a real verdict, but on a specific question: which method a team with limited compute reaches for first. It is not evidence that DPO and PPO produce equally good models.
PPO still leads where answers leave the data
The strongest challenge to the parity claim came in 2024, when a team led by Shusheng Xu ran DPO and PPO head to head on harder tasks and asked why they differ.6 The argument starts with the tether. PPO checks the model against the reference as it samples its own answers during training, so it is pulled back whenever it drifts. DPO has no such on-policy check: its loss can be driven low by a model that piles probability onto answers never seen in the preference data, as long as it ranks the seen pairs correctly. When the preference data sits off the model's own distribution, that gap shows up as worse behavior.6
On CodeContests, a competitive-programming benchmark, PPO tuning of a 34-billion-parameter code model solved 22.4 percent of held-out problems, above the 16.4 percent of AlphaCode, a system built for the task. DPO tuned the same base model to near zero.6 When human raters compared the two methods on a helpfulness set, they preferred PPO's answers to DPO's 45 times to 29.6 On tasks where an answer either runs or does not, PPO's leash to the reference model earns its cost.
This is not a clean reversal. On summarization, where the answers stayed close to the training data, DPO led PPO outright.1 The gap opens on the hard tasks, where answers can drift off the model's own distribution, and it is there that PPO keeps its edge.6
The takeaway
DPO's core claim holds. You really can replace the scoring model and the reinforcement-learning loop with one classification loss on ranked pairs, and it is much easier to run. That is why open models adopted it, and Zephyr beating a model ten times its size shows the payoff was real. The weaker claim is the slogan's other half. "As good as PPO" rests, in the original paper, on three small models, win rates judged mostly by another model, and a hardest task where no working PPO baseline existed to compare against. Later work fills that gap in PPO's favor: on problems where answers can drift away from the training data, PPO still wins, sometimes by a wide margin. DPO made preference tuning cheap and simple, and that is most of why open models run on it. It did not show that the cheap version always matches the expensive one, and on the hardest tasks it does not.
- 01
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model: the paper itself, with the derivation this lesson kept in plain words.
- 02
- Is DPO Superior to PPO for LLM Alignment?: the 2024 study that measured where PPO still wins and why.
Sources
- arXiv · Rafailov et al., "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (2023)
- arXiv · Ouyang et al., "Training language models to follow instructions with human feedback" (2022)
- arXiv · Christiano et al., "Deep reinforcement learning from human preferences" (2017)
- arXiv · Tunstall et al., "Zephyr: Direct Distillation of LM Alignment" (2023)
- Cameron R. Wolfe · "Direct Preference Optimization"
- arXiv · Xu et al., "Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study" (2024)