The model that drives safety to zero needs a lead nobody has held

A runner-up lab's choice becomes a formal model

Picture a lab in second place, a year into a research push it thinks can close the gap with the leader. Its board will not fund another year without a working demo, and its researchers track the leader's release calendar the way a following ship tracks the wake ahead of it. Spend the extra months on safety testing, and the lab may publish late and lose the funding round. Match the leader's pace instead, and it ships on schedule, whatever it did not have time to catch.

That is the choice Stuart Armstrong, Nick Bostrom, and Carl Shulman built into a formal model in 2016, generalized to any number of teams.¹ Each team has a capability level: how good its underlying research program is, drawn at random from a range the paper calls µ. A small µ means the teams are bunched close together in ability. A large µ means some teams can pull far ahead on capability alone, whatever they do about safety.

Each team also picks a safety level between zero, no precautions, and one, full precautions. Safety costs time: spending more of it means moving slower. A team's score is its capability minus its safety spending, c - s, and the highest score wins. The winner's system then works with probability s, its own safety level. Otherwise, with probability 1 − s, it causes what the paper calls an AI disaster. Losing costs something too, set by "enmity," from 0 (indifferent who wins) to 1 (a rival's success is as bad as disaster).¹

The paper solves for the Nash equilibrium, the choices where no team could improve its result by changing only its own, holding everyone else's fixed. Take the simplest case: two teams, no information, enmity at its maximum.¹

At µ = 0.5, equilibrium safety is 0.25. At µ = 0.05, it drops to 0.025. The paper states the general result plainly: "around µ = 0 (i.e. when capability is nearly irrelevant to producing the first AI), the only Nash equilibrium is to take no safety precautions at all."¹ When finishing first buys almost nothing on capability alone, every team's rational choice is to spend nothing on safety, even though every team would prefer, collectively, that none of them did. The model also assumes just a few comparable teams.

Knowing everyone's capability is worse than knowing none of it

The equilibrium above assumed no team knows anyone's capability, including its own. Armstrong, Bostrom, and Shulman also solve two more informed versions of the same setup. The ranking across all three is the paper's most surprising result: "The no-information case is always safer than the other two cases... It is always better if none of the teams have any idea about anyone's capability."¹

Under private information, a team that knows it is ahead has nothing to gain by cutting corners, since its capability alone already wins. A disaster then requires every team, independently, to have drawn low capability, a probability that shrinks fast, at the rate 1/µⁿ. Under public information, the leader cuts corners whenever the runner-up is close behind, since a close race is exactly when skimping on safety can decide the winner. A near-tie between the top two happens far more often than every team independently drawing low capability, so the danger shrinks only as slowly as 1/µ.¹

Plenty of people reject the racing framing's real-world premise, covered below. But no technical critique addresses whether this ranking survives once the model's assumptions relax, in a repeated game, say, or with capability that changes over time. The information result is the model's own claim; its robustness outside this exact setup is untested, not settled.

Every measured capability lead has closed within months

The model's danger rests on one assumption under the score itself: that finishing first is worth enough to justify cutting safety for it. That means a payoff close to winner-take-all, where a rival's success can be nearly as costly as a disaster. Enmity at its maximum, e = 1, is the model's own name for that stake, a narrower version of the decisive, discontinuous capability jump this desk tested separately.

Epoch AI's own Capabilities Index has not recorded a lead that lasted. The US-China frontier gap has averaged seven months since 2023, ranging from four to fourteen.² The closed-open gap has run three to four months.³ And OpenAI's o1, released September 2024, was matched by DeepSeek's R1 about four months later, "exhibiting mastery of the new paradigm."⁴

None of this shows a lead lasting long enough to become the exclusive advantage the model assumes. Every gap on record is a head start measured in months, not a permanent edge, and the open-closed gap has, if anything, widened slightly rather than closed to zero.³ This is the assumption the record contradicts most directly.

Testing time did shrink, just not to the "days" the headlines claimed

The model's second assumption, that safety trades directly against speed, has a genuine admission from at least one developer. OpenAI's Preparedness Framework, updated April 2025, names this exact scenario. If a rival ships a system with dangerous capability "without instituting comparable safeguards," OpenAI says it may adjust its own safeguards to match. It commits first to keeping them "more protective than the other AI developer," precisely "in order to avoid a race to the bottom on safety."⁵ That is one developer's account, not proof the field bends to it.

One independent evaluator has an on-record account of that pressure. METR, brought in by OpenAI to test o3 and o4-mini ahead of their April 2025 release, states its "work reported here was conducted over only three weeks." It warns that "more thorough evaluations are likely to reveal additional capabilities."⁶ Compare that with GPT-4's own account of roughly seven months of evaluation before its March 2023 launch.⁷

Reporting traced to a Financial Times investigation, read only through the outlets that carried it, puts the shrinkage further: "The company used to allow months for safety testing, but now gives just days."⁸ That figure is relayed at second hand, attributed to unnamed sources, and is more dramatic than METR's own three weeks for a related but not identical claim. The three weeks carries a named, on-record source. Treat "days" as reported, not established.

Today's commitments all take the same if-then shape

Most people who invoke this model today want developers to commit in advance to what they will do if warning signs appear, not to prove a disaster is coming. At the 2024 Seoul summit, sixteen companies signed a UK-brokered commitment, joined by four more the following year. Signatories agree to "set out explicit processes they intend to follow if their model or system poses risks that meet or exceed the pre-defined thresholds," up to not deploying "at all" past that threshold.⁹ This desk has covered a version of this if-then proposal from inside individual labs.

Holden Karnofsky, then a visiting scholar at the Carnegie Endowment for International Peace, made the same case more concretely that year: commitments of the form "if an AI model has capability X, risk mitigations Y must be in place." His conclusion: voluntary versions of this "alone are unlikely to keep risks low."¹⁰

The starkest ask came earlier. In March 2023, the Future of Life Institute's open letter was signed by over 30,000 people, including Yoshua Bengio, Stuart Russell, and Elon Musk. It called on "all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4."¹¹

Two serious critics reject the premise these proposals share. Yonathan Arbel and Matthew Tokson argue that AI knowledge is leaky, "rapidly copied, distilled, and reverse-engineered," so no lead lasts long enough to justify racing for it, matching the fast-follow pattern shown above.⁴ Paul Scharre rejects the label on more technical grounds. A classical arms race, he argues, needs defense spending growing 10 to 25 percent a year, a pace military AI spending has not matched. He calls the underlying concern "real but driven by perception, not inevitability," while granting, citing the V-22 Osprey program, that schedule pressure causes incomplete testing.¹²

Both camps agree that competitive pressure exists and can degrade safety in principle. Where they split is whether the model's actual stakes, a payoff big enough that a rival's win is nearly as costly as a disaster, are on the table at all. The commitments above are what signatories say they will do, not proof any has been tested. The dismissers' point, that leads do not last, is the best-evidenced claim either side has. Confidence runs ahead of proof, on both sides, over how likely a genuine, winner-take-all race is to occur before fast-following catches up.