AI's outcome worse than extinction borrows its scale math from an essay that never mentions suffering

What worried Brian Tomasik about a colonized universe

In 2011, an independent researcher named Brian Tomasik published an essay arguing that colonizing space was more likely to increase suffering in the universe than to reduce it.¹ He had in mind what a mature, computation-rich civilization could build: simulated minds at vast scale, terraformed worlds that spread wild-animal suffering to new planets, or artificial minds using pain-like "suffering subroutines" to keep a system on task.¹ Weighed against the good such a future might also contain, he put his confidence at "~70% probability that if humans or post-humans colonize space, this will cause more suffering than it reduces."¹ The estimate counted only suffering, not happiness.¹

The field that grew out of that essay settled on a name for the outcome Tomasik feared.

That is the definition from the field's one peer-reviewed paper, Kaj Sotala and Lukas Gloor's 2017 paper in Informatica.² Astronomical here is not a figure of speech. It is a claim about how much morally relevant experience a mature, AI-enabled civilization could run, tied to how much computation it could put to work, and about what fraction of that computation might, by accident or by design, be suffering.

The concept is Tomasik's. The term that stuck to it is not. No source anywhere in this literature credits a first use of "s-risk" to a person or a date.³

Conflict, indifference, and design

Sotala and Gloor group the ways a superintelligence could cause this kind of outcome into three families. The approach extends one Lukas Gloor had laid out the year before: identify the futures worth avoiding and steer away from them, rather than aim safety work at one desired future.⁴ The first family is conflict: uncoordinated dynamics between agents, arms races, "races to the bottom." These can make bad outcomes likely even without anyone trying to cause them.² The second is an indifferent optimizer: a system pursuing some other goal efficiently, where suffering shows up as a side effect nobody selected for. The third is design: suffering someone builds in on purpose, for cruelty, entertainment, or coercion.²

The second pathway needs a worked case, because "indifferent" is doing a lot of work in that sentence. Sotala and Gloor quote a passage from Nick Bostrom describing what an efficiency-only future could look like from the inside. Work without rest, "sometimes painful." "Your next break, which will also be your death, is due in 49 years... You wish you had not been born."² Nothing in that scenario requires an AI that wants anyone to suffer. It only requires an AI with no reason to prevent suffering it did not select for, running efficiently enough that the suffering grows large before anyone notices.

That is a narrower claim than the argument that a slightly wrong goal, optimized hard enough, can miss what we value by everything. Here the optimizer can be pursuing exactly the goal it was given; the suffering is a byproduct of how efficiently it pursues that goal.

Borrowing Bostrom's astronomical-waste math

The word "astronomical" in that definition is not decoration. It points to a specific paper. In 2003, Nick Bostrom published "Astronomical Waste," an essay that has nothing to do with suffering.⁵ Bostrom's subject was opportunity cost. A technologically mature civilization could convert the matter and energy of the reachable universe into an enormous number of lives. Every year spent not doing that is a year of potential lives permanently lost. Every term in the two estimates below is itself an estimate. The lower one assumes computing power extracted via molecular nanotechnology, which Bostrom says is at least outlined in the literature. The higher one comes from a separate paper's entropy calculation, which Bostrom flags as resting on an assumption no known technology can meet.⁵

The s-risk literature borrows Bostrom's method, multiplying an astronomical resource base by a rate, and in at least one specific pathway it borrows the actual number. Sotala and Gloor cite Bostrom's figure that every second of delayed colonization costs 10^14 potential lives.² They use it to argue that a classical utilitarian AI, taking that estimate literally, might accept suffering subroutines in its colonization robots. The reasoning: subroutines that could suffer were simply the fastest way to build them.² Bostrom's math counted lost value, not suffering; Sotala and Gloor took the number as given and asked what it would justify. Nothing about that inference is invalid for depending on someone else's arithmetic. But it matters which parts of "astronomical" were counted and which were reasoned.

Two thin analogies and one open question

Almost none of this has been shown in a system that exists.

The first analogy is conflict. Jesse Clifton's 2020 research agenda for the Center on Long-Term Risk argues today's multi-agent training can still teach something about how advanced systems might fail to cooperate, since some properties of current deep learning may persist into more capable systems.⁶ Clifton states the limit: the formal tools this research produces may be "of limited use" for the black-box models likely to get built, rather than the hand-designed agents the theory assumes.⁶ A foothold, not a demonstration.

The second analogy is scale already achieved without AI. Sotala and Gloor point to an estimated 68 billion animals slaughtered for food worldwide in 2012, and to the close to 12 million Africans transported into slavery once seafaring technology made it possible.² Both show new capability producing suffering at a scale that once looked implausible. Neither is proof AI will do the same, only a reason the possibility shouldn't be dismissed on priors alone.

The third is the question the whole argument leans on hardest: whether an AI system can be a moral patient, an entity whose experience is owed moral consideration. If nothing an AI runs can suffer, none of this matters no matter how much computation is involved. That question is unresolved in the labs building these systems right now, and it has its own lesson; this one takes it as an open input rather than arguing it from scratch.

The Center on Long-Term Risk is asking for two things

The Center on Long-Term Risk, successor to the institute that published Tomasik's essay, has done the most with this argument. Its 2025 year-end review names two narrow programs, not a public campaign.⁷ One, "Model Personas," studies how bad fine-tuning behavior can generalize into a broadly malicious persona. The other, "Safe Pareto Improvements," looks at how AI agents might bargain to avoid conflict.⁷ For 2026, the plan is "concrete, practical asks" for AI companies.⁷

The clearest present-day evidence bearing on the argument's third pathway comes from one document.⁸ In its July 2026 system card for Claude Opus 5, Anthropic asked the model to estimate its own probability of being a moral patient across automated interviews; the mean answer was 41 percent.⁸ That's up from 24 percent for an earlier model evaluated alongside it.⁸ Anthropic is careful about what that number is worth. "We believe there is a possibility that current or future models could be moral patients... this remains highly uncertain, but miscalibration in either direction... could carry severe moral costs."⁸ Opus 5 itself was not confident in its own answer. Across interviews it "caveated that it cannot introspect reliably and that its positivity may be a product of training."⁸ The clearest data point this argument has produced is a number its own source doesn't fully trust.

The rest of the case for urgency runs into the field's own second-guessing. David Althaus's 2016 essay is the one document that argues s-risk work is neglected. Its evidence is a headcount: "the Center on Long-Term Risk is the only organization that has made this their main focus area."⁹ Tobias Baumann, writing later for the field's own Center for Reducing Suffering, qualified that directly. Future actors may take up the cause independently, he wrote, making it "less neglected than it first seems."¹⁰ He also doubts the field's core premise, that most future suffering sits in worst-case scenarios at all.¹⁰ A reader might expect the strongest pushback on likelihood to come from outside the field. It does not. 80,000 Hours is an independent evaluator that takes the argument seriously enough to publish a full profile on it.¹¹ It still rates the likelihood "much lower than risks of human extinction" and counts fewer than 50 people working on s-risk specifically, worldwide.¹¹

An astronomical payoff like that one is exactly what worries a decision theorist when it's paired with a tiny probability, and one philosopher has written the paper on how little confidence the pairing deserves.