When many can act alone, the most optimistic one decides

One optimist is enough to tip the group

Nick Bostrom, Thomas Douglas, and Anders Sandberg describe a situation they say "arises in many contexts." Several actors can each, on their own, carry out some initiative that would affect everyone. Each actor is well-meaning: she goes ahead only if she judges the initiative good for the common good, not for herself. Her judgment is her own, drawn from private information, and it can be wrong in either direction.¹

Because any one of them can act alone, the initiative goes ahead the moment a single actor judges it worth doing. The decision falls to whoever is most hopeful about it, not to what the actors believe on average. As the authors put it, "it is the highest estimate that will determine whether an initiative is undertaken, not the average estimate."¹ These actors are not rivals, which is what sets the case apart from a race between labs. Nobody is trying to beat anyone. The trouble comes only from the statistics of many independent guesses.

Now suppose the initiative is actually a bad idea, its true value to the common good negative. The group still acts as long as one actor's error is large enough to push her private estimate above zero. And the more actors there are, the likelier it becomes that at least one of them overshoots that far. With five actors whose errors are symmetric around the truth, the initiative is still carried out about half the time even when its true value sits near minus one. That is well into the range where every actor, on average, can see it is a mistake.¹

The counting is what bites. Hold the true value at minus one and the chance that someone acts wrongly passes one-half once there are four actors, then climbs toward certainty as more join.¹ This is the unilateralist's curse: a group of well-meaning actors, each deciding alone, goes ahead too often, and no one in it has done anything unreasonable by her own lights. The authors' proposed fix is a "principle of conformity." Each actor lowers her own chance of acting alone to the level that would cancel the group's bias. She can share what she knows. She can revise her estimate down on the reasoning that being the most eager actor is itself a sign she has overshot. Or she can defer to a vote or an institution rather than her own first call.¹

Four assumptions hold the proof together

What Bostrom, Douglas, and Sandberg have is a piece of decision theory, and it is correct inside its own setup. That setup rests on four assumptions. The actors decide independently. They all want the same thing, the common good, and nothing for themselves. Their errors are private and symmetric, as apt to be too cautious as too reckless. And they are naive, each acting on her first estimate rather than reasoning about what the others will do.¹ Each of the four is also a place where the jump from the theorem to a real model release can come apart.

The last row is the one the authors press hardest themselves. They show the curse is a curse for naive actors in particular. A sufficiently careful actor can "manage the curse even without communication" by conditioning on the very fact that she is the one about to act, since being the most eager is itself a reason to trust her own estimate less.¹ So the result is strong for actors who take their first estimate at face value and weak for ideal reasoners who correct for the setup. Real labs sit somewhere between the two. That gap is exactly where the argument stops being a proof about AI and becomes an analogy to it.

A more hopeful actor undid GPT-2's caution

The clearest real disclosure with this shape is GPT-2. In February 2019 OpenAI withheld the full 1.5-billion-parameter model and put out smaller versions first, warning that the full one could be misused. The caution rested on the model being hard to reproduce. That premise did not hold. In August 2019 two master's students, Aaron Gokaslan and Vanya Cohen, trained a 1.5-billion-parameter model of their own and released the weights. OpenAI released its full model that November.²³

1. Feb 2019

   OpenAI withholds the full GPT-2²

   The 1.5-billion-parameter model is held back on misuse grounds; smaller sizes ship first as a staged release.

2. Aug 2019

   Two students replicate and release it³

   Gokaslan and Cohen rebuild a model of the same size and post the weights, months before OpenAI's own plans.⁴

3. Nov 2019

   OpenAI releases the full model²

   The public 1.5-billion-parameter model was set by the most release-inclined actor, not the most cautious one.

That is the curse's shape in a real case: the group outcome, a public model of the size OpenAI held back, was fixed by whoever was keenest to release. It is a shape, not a proof. The students' motives and the way their judgment tracked OpenAI's are not the independent, purely public-spirited estimates the theorem assumes.

The larger present-day version is open-weight release of frontier models. Meta put Llama 2 up for download in July 2023, when rivals kept comparable systems behind paid interfaces, though its licence carries restrictions and its training data stayed undisclosed.⁵ A year later it released Llama 3.1, including a 405-billion-parameter model, while OpenAI, Anthropic, and Google kept weights of that scale closed. Mark Zuckerberg's stated case is that open weights are safer because more people can inspect them. He adds that withholding does not keep a capable model from a determined adversary, since "stealing models that fit on a thumb drive is relatively easy."⁶

The governance work that argues against open release reasons in this same structure without using the name. Seger and about two dozen co-authors at the Centre for the Governance of AI conclude that some highly capable models "should not be open-sourced, at least not initially," and propose deliberation and government authorization to "dissipate control away from unilateral decision-makers," which are the paper's own remedies pointed at releases.⁷ Shevlane and Dafoe turn the publish-or-withhold question on "counterfactual possession," whether an adversary would obtain the capability anyway.⁸ Neither invokes the curse by name. The named application to AI is scarce, and the sharpest one comes from a critic of the remedy.

Existing institutions may already handle it

Stuart Armstrong, a colleague of the authors, granted that the result "has the advantage of being true" and then argued its practical reach is narrow, because society already corrects for it. International agreements, regulation, and markets force actors to share information and overrule lone optimists across most domains, and those systems are built to work without every actor being individually rational.⁹

He also pressed the harder point that acting alone is often good. Entrepreneurship and scientific progress depend on people pursuing projects the majority thinks worthless, so the curse "may thus be a valid individual warning for individual entrepreneurs" while it "fails as an overall judgement of the entrepreneurship."⁹ The authors concede as much. They note that "what is now widely recognized as important progress was instigated by the unilateral actions of mavericks," and that where a group is already too timid, more conformity makes it worse.¹ Pressed too far, deferring to the aggregate becomes a veto for the most cautious member.

There is a second edge. When others will reproduce or release the capability regardless, one actor's restraint buys little, which is the counterfactual-possession point put plainly, and GPT-2 is the worked case: the caution did not hold once someone else could act. Armstrong locates where the curse still bites hardest, in tail risks that are new and under-regulated, and names future rather than current artificial intelligence as that kind of case. Even there, he adds, the lesson is caution and not prohibition: an actor "shouldn't rely on their own naive initial judgement," which is a long way from telling her not to act.⁹