What Could Go Wrong

AI is getting measurably better at building AI

The sixty-year-old case for a runaway "intelligence explosion" rests on a real, doubling trend and on one loop that no working system has ever been seen to close.

8 min read2026-07-23

Why this matters

You keep meeting two confident claims about AI. One says a takeoff to superhuman intelligence is close, maybe a few years off. The other says the whole idea is science fiction and today's models are glorified autocomplete. This lesson hands you the single distinction that lets you judge either one on its evidence. It is the line between what a working AI system has been measured doing and what is only extrapolated about a system that does not exist yet. The argument is called the intelligence explosion, it is sixty years old, and it is driving real decisions right now about how AI gets governed. By the end you will know which parts of it rest on a measurement and which rest on a forecast.

Background optional reading

01: Models fake alignment when handed a goal, not yet on their own: the sibling argument about misaligned goals at a fixed capability, which is a different axis from the one here.
02: The record on reward hacking runs out before the catastrophe: the same move examined here, a measured pattern stretched past its evidence.

Why a cleverer machine would build a cleverer one

In 1965 the statistician I. J. Good wrote the sentence that has set the terms of the argument ever since. Let an ultraintelligent machine be one that far surpasses every intellectual activity of any person, he proposed. Designing machines is one of those activities. So such a machine could design an even better machine, which could design a better one still, and there would be, in his words, an "intelligence explosion." Good called the first such machine "the last invention that man need ever make." He added a condition in the same breath, one usually dropped when the line is quoted: "provided that the machine is docile enough to tell us how to keep it under control."¹

The idea is a feedback loop. A system good at building intelligent systems turns that skill on itself, and each turn produces a better builder, which turns the crank faster. Forty-five years later the philosopher David Chalmers set the loop out as a chain of plain steps. There will be human-level AI. If there is human-level AI, there will soon be something past it. If there is something past it, there will soon be something far past it. He named the assumption the chain leans on the proportionality thesis: that a gain in intelligence yields a proportionate gain in the ability to design intelligence.² If that holds, the steps do not stop at any human bound. A machine as far beyond us as we are beyond a mouse becomes a matter of enough turns. Serious people have found this loop plausible for six decades, because each step is hard to deny on its own.

The whole case is a single number

The most useful move anyone has made was to reduce the argument to one quantity. In a 2013 analysis, Eliezer Yudkowsky reframed the question as returns on cognitive reinvestment: when a system spends its intelligence on improving its own intelligence, how much does it get back? He borrowed the picture from a nuclear pile, where a chain reaction runs on a multiplication factor, the number of new fissions each fission causes. Below one, the reaction dies out. Above one, it runs away. The intelligence explosion is the claim that cognitive self-improvement has that factor above one.³

Everything turns on that number, and the defenders know it. Yudkowsky's evidence that it sits above one comes from evolution. It did not take ten times as long to get from Homo erectus to us as it took to reach Homo erectus, so the returns on improving a brain do not fall off a cliff. A machine, unlike evolution, can reinvest its gains directly in faster hardware and better algorithms, so its curve should climb steeper than ours.³ Nick Bostrom later sorted the possible speeds into a taxonomy the field still uses: a slow takeoff over decades, a moderate one over months or years, or a fast one over minutes, hours, or days. He judged the fast and moderate cases the likely ones. On the growth model Bostrom worked from, intelligence doubles in about seven and a half months and passes a thousandfold in under eighteen.⁴

The defenders named the weak joint themselves

The argument's most careful defenders wrote the strongest objection into their own papers. Chalmers listed the ways the loop could stall and singled out one as "perhaps the most serious": diminishing returns. If a ten percent gain in intelligence buys only a five percent gain next time, then 2.5 percent, the sum converges and the explosion never arrives.² Yudkowsky raises the same worry in his own words before answering it. For most hard problems in computer science, he notes, it takes exponentially more computing power to win a linear gain in performance, because the easy gains get used up first.³

This is not a fringe complaint. It is the best-documented pattern in the economics of research. As the easy discoveries are made, the next ones cost more. The philosopher David Thorstad, arguing against the whole hypothesis, points to the record. Holding Moore's law steady from 1971 to 2014 took an eighteen-fold collapse in the productivity of semiconductor research. The number of new drugs approved per billion research dollars fell from more than forty in the 1950s to under one in the 2000s. Good ideas get harder to find, and Thorstad's key point is that this is a property of the problems, not of the people, so hiring a machine researcher does not repeal it.⁴ The defenders reply that a machine might still outrun the decline. Whether it can is the live question, and it is quantitative, not rhetorical.

What the systems in front of us actually do

The philosophy can be set beside the record of working systems, which is narrower and firmer. The clearest measurement comes from METR, a nonprofit that tests frontier models. It asks how long a task, rated by how long the job takes a person, a model can finish on its own. A model that in 2019 could handle jobs of a few seconds could, by early 2025, handle software tasks of about fifty minutes at a fifty percent success rate. The length it can manage has been doubling roughly every seven months.⁵ In human terms, GPT-3.5 could do tasks that took a programmer about thirty seconds; GPT-4 pushed that to about four minutes.⁶

7 modoubling of task length

50 minlongest task at 50% success

30 s → 4 minGPT-3.5 to GPT-4

That trend reaches AI research itself. On METR's RE-Bench, which pits language-model agents against human machine-learning experts on real research-engineering tasks, the agents already win at short budgets: given two hours, the best agents outscore the human experts. But the ranking reverses at longer budgets. Give both thirty-two hours, and the average human scores about twice the best agent. The benchmark's authors are blunt about the limits. Real research runs on ambiguous goals and feedback loops that can take months, and their tasks have none of that.⁷ So the demonstrated fact is real and narrow. Machines measurably do some of the work of building machines, and they are getting better at it every year. This is why the argument has moved from workshops to governments. In one 2026 survey, twenty of twenty-five AI researchers ranked automating AI research among the most severe and urgent risks they could name.⁸

The one thing no one has watched happen

The two halves sit side by side. On one is a measured trend in what models can do. On the other is the claim the argument actually makes: a sustained, self-feeding loop that accelerates on its own, without a human turning the crank. No one has seen the second thing. The International AI Safety Report, an independent review written for governments by a panel of researchers, says so plainly. There is, it finds, "minimal empirical understanding of feedback loops from AI automating its own research and development."⁹ Every generation of the loop past the first is extrapolation. A doubling in task length is data. A takeoff is a forecast.

What the people who make the argument today ask for follows from that gap. They want the loop watched for: evaluations aimed squarely at AI-research-automation capability, and if-then commitments that pre-agree what happens when a model crosses a line. The same report frames the danger as running in two directions at once.⁹ Underrate the trend, and a real, measured capability catches policymakers flat-footed. Overrate it, and they brace for a loop the evidence has not shown exists. The survey that put twenty-five researchers on record found them split along a fault line worth knowing: those inside the frontier labs leaned toward fast, explosive scenarios, and the academics leaned skeptical.⁸

What holds up

The length of tasks models can finish has doubled about every seven months since 2019, on an independent measurement.
Frontier agents already outscore human experts at short AI-research tasks.

What to be careful about

The advantage reverses on longer tasks, and the headline figure is at a fifty percent success rate.
No sustained, autonomous self-improvement loop has ever been observed.
Research productivity falls as the easy discoveries get used up.

The takeaway

The intelligence-explosion argument has one strong claim and one missing piece, and they are easy to tell apart once you look. The strong claim is that machines can improve machines, and the measured trend backs it. The length of tasks models can do has doubled about every seven months, and agents already beat human experts at short AI-research jobs. The missing piece is the loop itself, a self-feeding acceleration that runs without us, which no working system has been observed to do. So when you hear that takeoff is imminent, ask which system shows the loop, and you will find a forecast. When you hear that it is impossible, ask about the doubling, and you will find a measured trend that dismissal ignores. The number is real. The runaway is still a claim.

Go deeper optional reading

01: METR: the task-horizon and RE-Bench measurements behind the trend, updated as new models land.
02: International AI Safety Report 2026: the independent synthesis of where the evidence now stands.
03: Thorstad, "Against the Singularity Hypothesis": the case that diminishing returns sink the growth assumption.

Why a cleverer machine would build a cleverer one

The whole case is a single number

The defenders named the weak joint themselves

What the systems in front of us actually do

The one thing no one has watched happen

Sources