Bostrom says only luck has kept us from a world-ending technology

The black ball nobody has drawn

Making a nuclear weapon takes several kilograms of plutonium or highly enriched uranium, both so difficult and expensive to produce that the bomb has stayed in the hands of a few states. Bostrom asks a colder question: what if it had turned out otherwise? Suppose the physics had allowed anyone to release the energy in an atom the easy way, say by sending a current through a metal object placed between two sheets of glass.¹ The bomb would then need no rare material and no state program, only some glass and wire, and the first person to work it out could destroy a city. We do not live in that world. Bostrom's point is that nothing told us in advance that we would not.

He turns this into a picture. Human invention is like pulling balls from a giant urn. Most come out white, plainly beneficial; some are gray, mixed or moderately harmful.¹ What we have never pulled is a black ball: a technology that by default destroys the civilization that invents it. The easy-nukes world would have been a black ball. We have not drawn one, Bostrom writes, and the reason is not that we have been careful or wise. "We have just been lucky."¹

That sets up the claim itself. The Vulnerable World Hypothesis holds that if technological development continues, we will at some point reach capabilities that make the devastation of civilization extremely likely, unless the world sufficiently changes how it is organized.¹ Devastation is given a fixed bar: at least 15 percent of the world's population dead, or global output cut by more than half for over a decade.¹ Bostrom is careful about how much he is asserting. He does not claim a black ball exists. He calls the hypothesis's truth "an open question" and does not try to prove it.¹ The hypothesis is a warning about what one more draw could hold.

The condition that turns one bad draw into a catastrophe

A dangerous invention is not the same as a catastrophe. We know how smallpox works and still contain it. Bostrom's argument needs a second piece: a claim about the kind of world a black ball would be drawn into. He calls that world the semi-anarchic default condition, and defines it by three features.

Put those three together and a single black ball is enough. If a technology is destructive and easy to use, someone in the apocalyptic residual will eventually use it, and no state in the default condition can stop every one of them in time.¹ Not every black ball works through a lone actor. Bostrom sorts them by mechanism.¹ Some empower one person, as easy nukes would. Some hand a powerful state an incentive it cannot resist, such as a safe first strike. Some come from many hands each doing a little damage that sums to ruin, the shape of worse-than-expected warming. A last kind carries a hidden risk that fires the first time the experiment is run. The common thread is that the default world has no reliable brake.

What today's systems have actually shown

When people apply the hypothesis to AI now, the black ball they have in mind is usually a model that hands a lone actor the design for a pandemic pathogen or a mass cyberattack. So the fair test is what the systems in front of us have been measured to do. Start with bioweapons. In 2024 RAND ran a controlled red-team study: teams drafted operational attack plans, some with only the internet, some with a large language model added, and independent graders scored each plan on a nine-point viability scale. Access to a model moved the average score by 0.22 points in the wrong direction for the fear, a difference with a p-value of 0.64, far short of significance, and no plan in the study was judged viable.² The models tested were those available in the summer of 2023.

The frontier moved after that. In 2025 Anthropic ran its own uplift trial on newer models and reported a figure that climbs from one generation to the next.

OpenAI, testing GPT-5, decided to treat it as "High capability" in the biological domain and switched on the matching safeguards, while stating that it did "not have definitive evidence that this model could meaningfully help a novice" cause severe harm.⁴ Read the numbers to the halfway point and the trend looks alarming. Read the same documents to the end and the ceiling appears. Anthropic's 2.53x sits below the 5x it had fixed beforehand as the mark of "significant additional risk," and it calls text-based trials like it "substantially weaker proxies for real-world scenarios."³ Both labs raised their guards as a precaution, on rising numbers they themselves describe as weak evidence, not on any attack.

Cyber tells a cleaner version of the same story, because its strongest demonstrated results are defensive. Google's Big Sleep agent found a real, previously unknown memory bug in SQLite that standard fuzzing had missed, and it was reported and fixed the same day.⁵ In DARPA's 2025 AI Cyber Challenge, autonomous systems read more than 54 million lines of code and patched dozens of the vulnerabilities they found, including real bugs no one had planted.⁶ Every one of those results came from a team whose job was to fix what it found. None of it measures the thing the hypothesis fears, an attacker's cost to cause mass harm falling toward zero.

This desk has worked through both records in full, the bioweapon tests and the cyber ones. The line they draw is the one the hypothesis turns on. What a working system has shown is a measured uplift on a text proxy that is still rising, still under the labs' own risk lines, and still attached to plans that fail. What stays projection is the black ball itself, a model that closes the distance from a failed plan to a finished weapon. No system has drawn it, and no measurement yet shows one about to.

The remedy is where the fight is

Take the hypothesis seriously and the question becomes what to do. Bostrom's answer is the part that has drawn the most fire. To make a black ball survivable, a society has to leave at least one of the three features behind. The drastic remedy is to end limited preventive policing: watch everyone, everywhere, closely enough to catch the one actor before they act. Bostrom spells out what that would take.

He prices it out. At roughly US$140 a person each year, the whole world could be watched continuously for less than 1 percent of global output.¹ Where watching is not enough, he floats preemptive incarceration, holding someone whenever unreliable signs put the chance above 1 percent that they will attempt a city-destroying act.¹ This is not a hostile caricature. It is Bostrom's own illustration of what stabilization might require. The sharpest published objections land here, on the cure rather than the diagnosis.

Set the two sides against each other and the gap is symmetric. The alarmed case runs ahead of its evidence, the rising but weak proxies just described. The dismissive case runs ahead of its evidence too: a clean run of failed tests on today's models is not proof that no black ball is reachable, which is exactly Bostrom's point about luck. What the argument has surely done is force a real cost into view. If the hypothesis is right, the price of safety is a tag around every neck. If it is wrong, we will have argued ourselves toward that tag on the strength of a danger we could never measure.