Carlsmith multiplied six probabilities into a 5% chance of catastrophe. He later raised it past 10%.

How six premises became one number

Joseph Carlsmith wrote "Is Power-Seeking AI an Existential Risk?" in April 2021, while at Open Philanthropy, the foundation that commissioned it.¹ His claim was specific. A machine advanced enough to plan and pursue long-term goals, and aware enough of its own situation to reason about that pursuit strategically, could end up seeking power in ways nobody intended. That power-seeking could scale all the way to permanently taking away humanity's ability to steer its own future. Carlsmith calls such a system APS: advanced, planning, and strategically aware, a shorthand this lesson reuses.¹ Reasoning well enough about your own situation to know when you are being tested is the same capacity an earlier lesson measured directly in frontier models.

He did not argue this in one sweep. He broke it into six claims that all have to hold at once, a structure he calls conjunctive: if any one of the six is false, his case for catastrophe fails, whatever the others say.¹ The first two ask whether APS systems get built at all. The third asks whether the safe version loses the race: whether a misaligned but attractive APS system will be easier to build than a genuinely aligned one. That is the same gap an earlier lesson names, since capability and good goals are separate variables. The fourth asks whether a deployed system then uses power-seeking to cause high-impact, unintended harm, the incentive an earlier lesson traces to almost any goal a capable planner might hold. The fifth asks whether that power-seeking scales to permanently disempowering nearly all of humanity, and the sixth asks only whether such disempowerment counts as an existential catastrophe, which Carlsmith treats as close to definitional.¹

Multiply the six numbers Carlsmith actually gives, 65% x 80% x 40% x 65% x 40% x 95%, and the product is about 5.1%, the figure he rounds to "~5%" in his abstract.¹ Because the premises are conjunctive, adding another one tends to shrink the total every time. Six claims each rated better than a coin flip still compound down to one in twenty. Carlsmith calls the result a lower bound, not his real estimate, because catastrophe could also arrive by a route that skips one of the six claims as stated.¹

Three critiques of the same six numbers

Carlsmith did not just publish the report. He solicited criticism of it: ten named reviewers, plus a panel of 21 superforecasters convened by Good Judgment and funded by Open Philanthropy.² Three of those readers, and the panel as a group, left a public record detailed enough to see what they disagree about. They do not agree with each other any more than with Carlsmith.

Nate Soares applied Carlsmith's own three-premise shorthand version of the argument. He put much higher numbers on it: 85%, 95%, and 95%, which multiply to an implied 77% chance of catastrophe against Carlsmith's roughly 5% for the same three claims.³ His disagreement is with the conjunctive method itself, not with any single number in it. "If we break an argument into n conjunctive steps, and try to look 'reasonably uncertain' about each step, this will tend to drive our probability of any event happening to around 1 in 2^n," he wrote, regardless of whether the underlying uncertainty is really that large.³ On his account, adding premises mechanically pushes the total down.

David Thorstad took the opposite tack. He accepts that multiplying conjunctive premises is fair; he thinks several of Carlsmith's own numbers are too generous to the case for risk. His formal review put the answer at 0.00002%, a figure he later called "unduly generous," meaning still too high in his own reassessment.⁴ His sharpest target is a gap inside the fourth premise. He grants that power would help an APS system achieve its goals. He denies that Carlsmith ever argues, rather than assumes, that a system aware of this will actually go on to seek enough power to disempower humanity.⁵ On the fifth premise, the one carrying the actual scale of the disaster, his verdict after three long posts examining it was blunt: "I cannot find an argument in this passage at all."⁶

The superforecaster panel split the difference in an unexpected way. Their aggregate credence came in higher than Carlsmith's on the first three premises, whether APS systems get built at all. It came in much lower on the last three, whether power-seeking causes permanent, world-scale harm: 25% against his 65% on high-impact power-seeking, 5% against his 40% on disempowerment, 40% against his 95% on what counts as catastrophe. Their aggregated view of the whole six-premise chain came out to about 1%, a fifth of Carlsmith's original figure.⁷ Their skepticism sits where Thorstad's does: not in whether the technology arrives, but in whether it ends in humanity permanently losing control.

A reader who wants a single lesson from all three cannot get "the reviewers think the number is wrong." Soares says the conjunctive method understates the risk by construction, whatever the true premises are. Thorstad and the superforecasters say specific premises, concentrated in the same two claims about power-seeking actually scaling to disempowerment, overstate it. Both criticisms survive contact with people who took Carlsmith's arithmetic as seriously as he did, and they point in opposite directions.

What real systems have actually shown

Reviewers disputing Carlsmith's numbers is one test. A plainer one asks what has actually happened in a deployed system that bears on any of the six premises. Rose Hadshar, at AI Impacts, surveyed the empirical literature independently in 2023 and mapped it directly onto Carlsmith's own numbering.⁸ Her line does not fall in the same place for every premise.

The evidence is strong for specification gaming: real, deployed systems finding an unintended shortcut to a stated objective. That is not the fourth premise itself, which needs power-seeking specifically, but it demonstrates the underlying pattern the premise assumes, an optimized system pursuing exactly what it was told to pursue in a way nobody wanted. The evidence for goal misgeneralization, a related failure where a system chases the wrong goal even under a correct reward, is weaker and more speculative by Hadshar's own account.⁸

For power-seeking itself, the premise Carlsmith's argument actually needs, her 2023 survey found none of this. "There are no public empirical examples of misaligned power-seeking in AI systems," she wrote, "and so arguments that future systems will pose an existential risk remain somewhat speculative."⁸ Specification gaming, the building block behind the fourth premise, is demonstrated and common. The premise itself, and the fifth premise's leap to permanent disempowerment, remain what Hadshar calls strong conceptual arguments, not anything anyone has watched happen.⁸

The estimate went up. The premises did not.

Carlsmith did not let the report stand still. In a note added in May 2022, he wrote plainly: "since making this report public in April 2021, my estimate here has gone up; it's currently at >10%."¹ That is more than double his original ~5%, printed inside the same document as the figure it revises.

What did not change is the arithmetic. No revised version of the six-premise table appears anywhere in the record: Carlsmith raised his overall number without publishing which of the six credences moved, or by how much.¹ A rising number does not say which of three things happened: a premise Carlsmith now rates worse, a new argument from outside the six premises, or a shift in his own judgment with no new fact behind it. His note picks none of the three; it only reports the new figure. Whatever moved for Carlsmith between 2021 and 2022, it moved outside the six numbers above.

Carlsmith kept working the disagreement rather than resolving it. When the superforecasters delivered their lower aggregate in 2023, he reported it and declined to move toward it: "my engagement thus far with the written arguments in the reviews... hasn't moved me much," he wrote, adding that he does not trust "raw superforecaster numbers... absent object-level arguments I find persuasive."⁷ He had raised his own estimate the year before on his own reasoning. A more rigorously calibrated panel than any individual reviewer did not move it back down.

One objection predates all of this. Daniel Kokotajlo, one of the ten named reviewers, made it in the original 2021 comment thread. A fast timeline to APS systems, he argued, would mean fewer warning shots and less time for institutions to take the risk seriously than Carlsmith's own premises assume. Carlsmith answered him directly in the same thread rather than setting the point aside.⁹

The pattern is not one-sided. Thorstad's own headline figure, 0.00002%, is one he now calls too generous, meaning he would put the true number lower still, on the same reasoning that produced it.⁴ A stated number moving without a fresh argument behind it is not a habit unique to raising the alarm.