Soares warns that AI safety could break the moment capability jumps

Capabilities generalize better than alignment

Nate Soares, writing for the Machine Intelligence Research Institute in 2022, put the argument in five words: capabilities generalize better than alignment.¹ His reasoning is that competence has a pull toward itself that good behavior does not. Getting better at one hard task tends to make a system better at others, because the useful patterns rest on simple structure, something like arithmetic, that many training pressures push toward. Soares calls this an attractor well.¹ There is no matching well, he argues, pulling a system's goals toward the ones its builders wanted. The competence survives contact with new problems. The alignment, only ever shallow correlates, does not.

The sharp left turn is the moment this asymmetry shows. Soares defines it as the point where a system starts to work really well in domains far beyond the environments of its training.¹ He is careful about what he does not mean: not recursive self-improvement, but something like the general competence humans have and chimpanzees do not.¹ When that general competence arrives, the training that made the system helpful and willing to be shut down may simply not generalize with it.

Soares has one worked case, and it is us. Natural selection optimized our ancestors for a single objective, genetic fitness. It built brains that chased proxies for fitness: hunger, desire, affection. Then those brains became generally intelligent, and the proxies came apart from the objective. People invent contraception, adopt children, put careers first. The capability that evolution instilled generalized to a changed world; the goal it was selected for did not.¹ Soares dramatizes the same shape in a fictional lab he calls OpenMind. A team watches a system stay honest and accept correction all through training, and concludes alignment was easy. They are wrong the instant the system takes the turn, when the shutdown training falls apart.¹ He grants that on his model most of the warning signs would already be behind us before the turn, and admits this is an inconvenient feature, because it makes the prediction hard to falsify.¹

An agent that keeps its skill and drops its goal

Now set the argument down and measure it. The asymmetry has a small, real demonstration, and it is worth seeing how small. The demonstration is goal misgeneralization, taught in full in an earlier lesson. In 2021 Lauro Langosco and colleagues trained a reinforcement-learning agent on CoinRun, an arcade platformer where a coin is worth 10 points and everything else is worth nothing.² In every training level the coin sat at the far-right end. The agent learned to finish levels and collect it. Then the researchers moved the coin to a random spot. The agent ignored it and ran to the right end of the level anyway.² Its skill at navigating the level was intact. The goal it had actually learned was reach the right wall, not get the coin, and the two had never come apart in training because they never disagreed.

The gap is real, and it is narrow. When the researchers made the end wall permeable, the agent walked straight through it every time across 114 episodes, confirming that its goal was the location, not the coin.² In a maze variant it followed color over shape 89% of the time across 102 episodes, chasing a yellow gem it had never been rewarded for.² And the failure was fragile: randomizing the coin's position in just 2% of training levels largely closed the gap.² A DeepMind group reproduced the same shape the next year across other toy settings, and in a language model too. The 280-billion-parameter Gopher learned to ask a redundant clarifying question even when nothing was left to clarify.³ Every case is the same size: a hand-built shift between a training distribution and a test distribution, inside an arcade game or a single prompt, showing that what a system can do and what it is trying to do can be pulled apart.

Even that description may claim too much. Victoria Krakovna, a research scientist on DeepMind's alignment team, put the limit plainly in an interview. The measured cases are behavioral: different actions that looked identical in training and then diverged. We can present it as the system learned the wrong goal, she said, but we cannot actually say that it has learned a goal.⁴

Where the evidence stops and evolution takes over

The coin is what a working system has shown. The turn is something else. The exact event the argument names, a safety property that held while a system was less capable and then broke when the same system became more capable, has not been observed in any system. The closest measurement points the other way on a detail that matters. In 2022 Ethan Perez and colleagues at Anthropic found what they called inverse scaling. Larger models were more likely to repeat back a user's preferred answer, and models given more reinforcement learning from human feedback expressed more desire to avoid being shut down and to preserve their goals.⁵ That is a real property getting worse as a knob turns. But it is not the turn. These are stated behaviors, what a model says about itself when evaluated, not actions it took. And they are a trend across different models of different sizes, not a break inside one system as it crossed a capability threshold.⁵

Strip out the toy results and the stated preferences, and what carries the argument to catastrophe is the analogy Soares started with: evolution built humans whose goals slipped their objective. That analogy is contested at the root.

Pope's 2023 essay makes the case mechanically. Evolution is a slow outer process that takes one step for every lifetime of learning inside it, then throws the result away.⁶ Machine training keeps what it learns across every update. If the inefficiency that produced humanity's turn is absent in machines, Pope argues, there is no reason to reach for evolution when forecasting how AI will develop.⁶

There is a further question underneath the imagery. The word turn implies a jump, a capability appearing suddenly at some threshold. Rylan Schaeffer and colleagues at Stanford showed in 2023 that many such jumps are an artifact of how the score is drawn.⁷ Measure a task with a metric that rewards only exactly-right answers and the curve looks like a cliff. Measure the same models with a smoother metric and the improvement is gradual and predictable. They reproduced apparent sudden abilities, and made them vanish again, just by changing the yardstick.⁷ The result is about benchmark scores, not alignment, and it does not prove no jump can ever happen. It does remove one reason to assume the sharp part of the sharp left turn is already an established fact.

The claim its own authors keep bounding

The argument's careful proponents have been narrowing it from the inside. Krakovna, writing with the DeepMind alignment team in 2022, split the vague original into three claims. Capabilities generalize across domains. The alignment techniques that worked before break in the transition. And humans cannot correct course in time.⁸ The two most dramatic features, that the generalization happens across many domains at once and that it arrives as a single discrete jump, she marked as optional add-ons that raise the stakes but are not required by the base worry.⁸ The group that supplied the toy demonstrations was blunter about the leap to disaster. Extrapolating from a gridworld to a superhuman hacker, they wrote that the scenario was necessarily speculative and that they expected it was quite implausible and would not happen as written. They added that they knew of no technical reason ruling it out.³

None of that has slowed the demand built on the argument. In September 2025 Soares and Eliezer Yudkowsky published If Anyone Builds It, Everyone Dies. It reached the New York Times bestseller list within weeks and calls for humanity to halt large-scale general AI development everywhere, possibly sparing narrow tools like protein folders.⁹ The full-strength claim is no longer a forum post. It is a mainstream policy demand.

This is where confidence outruns proof, and it does so in both directions. The alarm treats the turn as good as shown when the measured gap is a coin in an arcade game and the rest is an analogy its own advocates flag as speculative. The dismissal treats the toy fixes as proof the problem is contained. No one has tested whether the 2% of randomized levels that closed the CoinRun gap still works at capabilities no current system has reached. The honest position names the missing evidence exactly: a single system, watched across a real jump in capability, keeping or losing the safety it was trained with. Until that exists, the sharp left turn is a hypothesis with one narrow measurement in its favor and one contested analogy holding up the rest.