DeepSeek trained a model to reason without ever showing it a worked example

A reasoning model trained without a single worked example

"DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning" is a January 2025 paper from DeepSeek-AI, later peer-reviewed and published in Nature in September 2025.¹² It documents two models. R1-Zero is the one that made the paper famous: DeepSeek trained it to work through math and coding problems in long chains of reasoning, the same step-by-step behavior this course covered as chain-of-thought prompting. It learned that behavior using reinforcement learning alone, without a single labeled example of what a correct reasoning chain looks like. On AIME 2024, a 30-problem math-competition benchmark, R1-Zero solved 71.0% of its problems on one attempt each.¹ DeepSeek released the weights for both models rather than keeping them behind an API. That choice is why a paper about a training method turned into an argument about whether frontier reasoning still required a closed lab and a nine-figure budget.

Both models sit on top of DeepSeek-V3-Base, a mixture-of-experts model with 671 billion parameters in total, 37 billion of which activate for any single token, pretrained on 14.8 trillion tokens.¹ Neither R1-Zero nor R1 was trained from nothing. What the paper actually tested was narrower and more specific: whether reinforcement learning, applied to a base model already that capable, could produce long-chain reasoning on its own.

Two training runs, each starting from the same base

The reinforcement learning behind both models is GRPO, Group Relative Policy Optimization, introduced in an earlier DeepSeek paper on math reasoning.³ GRPO is a variant of the reinforcement-learning algorithm PPO, redesigned to cut its memory overhead while improving reasoning performance. The "group" in the name is a batch of the model's own sampled answers to the same problem. GRPO scores each answer against that group's average rather than against a separately trained value model.³ DeepSeek first deployed it in DeepSeekMath-7B, where it pushed a 7-billion-parameter model to 51.7% on a competition-math benchmark.³

R1-Zero is DeepSeek-V3-Base run through GRPO with only two automatic checks on each sampled answer: is it correct, and does it follow the required format. No human-labeled reasoning steps, no supervised fine-tuning stage at all.¹ That run, 512 H800 GPUs for about 198 hours, cost $202,000.¹ Partway through it the model began pausing mid-answer to re-examine its own steps, a behavior the authors had not trained for directly and named the "aha moment" in the paper.¹ R1-Zero was not the model DeepSeek built to last. Instead it built a second model, R1, using R1-Zero's own output as raw material. Before any reinforcement learning, R1 is first fine-tuned on roughly 800,000 examples, built for $10,000 of compute.¹ 600,000 of those examples are reasoning traces, harvested by running the R1-Zero checkpoint repeatedly and keeping the correct, well-formed answers. The other 200,000 are ordinary instruction examples drawn from DeepSeek-V3's original pretraining data, which keep the model useful outside math and code. Only after that head start does R1 go through its own reinforcement-learning run, the same kind as R1-Zero's. That run, on the same 512 H800 GPUs for about four days, cost $82,000.¹ The paper reports all three stages in one table: R1-Zero's run at $202,000, the SFT data at $10,000, R1's own run at $82,000. They sum to a $294,000 total for the whole pipeline that turns the base model into R1.¹

Two numbers for every benchmark

DeepSeek's headline claim rests on a small set of tests, each reported two different ways. On AIME 2024, a single attempt is called pass@1: one try, one answer. R1-Zero solved 71.0% of its problems pass@1 and R1 solved 79.8%, a result the model card DeepSeek published alongside the weights repeats exactly.¹⁴ The paper also reports a second number for the same test: cons@64, short for consensus across 64 samples. The model answers the same problem 64 times, and the score counts a problem correct if the majority of those 64 attempts agree. Under that method, R1-Zero's AIME 2024 score rises to 86.7%, 15.7 points above its one-try score. The gain comes from asking the same question sixty-four times and taking a vote, not from any change to the model.¹ MATH-500, a 500-problem subset of a standard math benchmark, moved less between R1-Zero (95.9%) and R1 (97.3%) on a single attempt. Codeforces, a competitive-programming platform that rates solutions the way chess rates players, put R1-Zero at 1444 and R1 at 2029.¹

Those numbers describe what the paper reports the model did. They say nothing about whether the model had seen versions of the questions before, on this test or any other. A 2026 study tested six frontier models, DeepSeek-R1 among them, for contamination on MMLU, a different, general-knowledge benchmark: whether training data overlaps with test questions.⁵ On a 513-question sample, 13.8% of the questions themselves showed a contamination signature in a web search, a property of the question set measured before any model answered, rising to 18.1% in STEM questions and 66.7% in philosophy.⁵ Averaged across all six models, accuracy on those same questions fell 7.0 percentage points when they were reworded as indirect references instead of asked directly, and 19.8 points on law and ethics questions specifically, the shape of an answer partly remembered rather than derived.⁵ DeepSeek-R1 did not follow that average. Its own accuracy on the original questions was 29.2%, far below the other five models tested, and its accuracy on the reworded version barely changed, dropping about 3.2 points, the smallest drop of the six.⁵ R1's memorization signature was also unusual: 76.6% partial reconstruction of the source material but 0% verbatim recall, familiarity with MMLU's content rather than its wording.⁵ None of this tests AIME, MATH-500, or Codeforces directly. What it shows is a model that knows something about MMLU's questions without having memorized their answers outright, a different problem than the one the six-model average illustrates.

An outside lab reran the recipe from scratch

DeepSeek's recipe has a fourth step beyond the two training runs: distillation. Take reasoning traces produced by R1 and use them as ordinary supervised fine-tuning data for a smaller model, with no reinforcement learning required for that smaller model at all.⁶ Hugging Face's Open-R1 project, an independent reproduction with no stake in DeepSeek's own claims, rebuilt this pipeline from published information rather than DeepSeek's own code. In the process it released its own reasoning datasets: 350,000 verified traces called Mixture-of-Thoughts, a 220,000-problem math subset, and 100,000 competitive-programming solutions.⁶ Fine-tuning a 7-billion-parameter model on that data, Open-R1-Distill-7B reached 52.7% pass@1 on AIME 2024 and 89.0% on MATH-500.⁶ Both scores sit well below R1's own 79.8% and 97.3%, unsurprising for a dense model a small fraction of R1's 671-billion-parameter total size. But they track the relationship DeepSeek reported: reasoning learned through reinforcement learning on a strong base model transfers into a much smaller model through fine-tuning alone.

The bill that got swapped for a bigger one

The number attached to DeepSeek-R1 more than any other is $5.6 million, repeated as the cost of building a frontier reasoning model. The pipeline that actually produced R1's reasoning, R1-Zero's run plus the SFT data plus R1's own run, cost $294,000 in total, by the paper's own accounting.¹ Epoch AI, an independent analyst with no role in either paper, modeled the same training from public information and arrived at a higher number: roughly $1 million for the RL phase alone and about $6 million total. That estimate excludes personnel, administrative, and experiment costs, and it does not reconcile against DeepSeek's own disclosed figures at all.⁷ Where an independent estimate and a lab's own disclosed accounting disagree by roughly three to twelve times on the same undisclosed line items, the honest read is a gap, not a confirmation. The $5.6 million figure itself is real, and it prices a different project entirely. It is DeepSeek's own disclosed cost for pretraining DeepSeek-V3-Base, the 671-billion-parameter model underneath both R1-Zero and R1: 2.788 million H800 GPU-hours, $5.576 million. That figure comes from the separate technical report DeepSeek published on V3 itself, not from the R1 paper or its model card.⁸ Quoting "$5.6 million" for R1 swaps the base model's bill for the reasoning pipeline's. It understates what the base model underneath R1 actually cost by treating it as part of the same project, and it overstates what building R1's reasoning on top of that base model cost, by a factor of about nineteen. The same retelling folded "beat OpenAI" into the same confident tone as the cost claim, a framing DeepSeek's own paper does not make and that the contamination study above does not test, since it covers MMLU, not the benchmarks people actually cite.