Automated markets already break the way Perrow warned reactors would

A meltdown with no broken part

In 1984 the Yale sociologist Charles Perrow published Normal Accidents, written after the 1979 partial meltdown at Three Mile Island. Its argument was that some catastrophes are ordinary. In some systems, a disaster can be a normal property of how the system is built, with no single part having failed in a way anyone would call a disaster. Perrow named the phenomenon a normal, or system, accident: the "unanticipated interaction of multiple failures" in a system running a high-risk technology.¹

Two properties put a system in that danger. The first is interactive complexity. In a complex system the parts affect one another in ways an operator cannot foresee. A nuclear plant is complex in this sense: no designer fully maps how its thousands of components can interact. The second is tight coupling. A system is tightly coupled when one event follows another rapidly and almost invariably, with little slack to step in before the fault propagates. Where a system has both and runs a high-risk technology, Perrow argued a catastrophic accident is only a matter of time.¹

The corner where accidents are normal

Perrow's two axes give four combinations, defined here in his own words.

The danger sits in one corner, both complex and tightly coupled. Perrow placed high-hazard systems this way.

The theory covers less than the people who borrow it tend to assume, and Perrow drew the line tightly. In his longest chapter, on collisions at sea, only 5 to 10 percent of the accidents were system accidents in his sense; the rest were ordinary failures of a single component behaving predictably.¹ He later placed Bhopal, Chernobyl, the Challenger, and the Exxon Valdez outside the theory, calling them component-failure accidents and "alarmingly banal examples of organisational elites not trying very hard."² Stretching the term to AI failure in general claims more ground than he did.

The cascade that actually happened

So far the theory rests on nuclear plants and the like, where catastrophe is rare enough that inevitability is hard to check. The clearest case of the mechanism in a working system is elsewhere, an automated market. An earlier lesson covered the 2010 Flash Crash in full; what matters here is the shape of it.

At 2:32 p.m. on May 6, 2010, one mutual-fund complex began selling 75,000 E-Mini S&P 500 futures contracts, about 4.1 billion dollars, as a hedge. Its automated program sold at 9 percent of the previous minute's trading volume, "without regard to price or time."³ A comparable sale had once taken more than five hours. This one, into an already stressed market, finished in about twenty minutes.³ High-frequency traders bought the contracts, then, holding more than they wanted, sold them to one another. Between 2:45:13 and 2:45:27 they traded more than 27,000 contracts, roughly 49 percent of the volume, while buying only about 200 net.³ Buy-side depth in the E-Mini fell under 1 percent of that morning's level.³

The mechanism is tight coupling. Automated strategies responded to one another faster than any person could read the screen, let alone intervene. No trader in the loop had a misaligned goal, and the sell program was doing exactly what it was told. The interaction of well-behaved parts produced the cascade.³

Then the coupling was loosened. At 2:45:28 the exchange's automatic logic paused E-Mini trading for five seconds, and within about twenty minutes the market had largely reverted.³ Afterward, regulators added slack by design: single-stock circuit breakers that pause a stock for five minutes after a 10 percent move.³ The same day that shows tight coupling outrunning intervention also shows the system given slack once people understood it, which cuts against reading the cascade as inevitable.

The Flash Crash is also not the clean case of blameless parts it is sometimes told as. In 2015 the Commodity Futures Trading Commission charged a London trader, Navinder Sarao, with manipulation and spoofing that it said "contributed to" the order-book imbalance that day.⁴ The 2010 report still names the automated sell program as the trigger, and the cascade needed no bad actor to run, but an honest account keeps the manipulation in view.

Tightly coupled automation running faster than people can intervene is on the record. A catastrophic normal accident in artificial intelligence is not.

Aiming the theory at AI

Perrow kept making the argument late in his life. Writing about the Fukushima disaster in 2011, he called it "prosaic, even," one more case where flawed design, components, procedures, and operators combine as, he held, they always will. He pressed the political question his theory implies: "whether some industrial systems have such huge catastrophic potential that they should not be allowed to exist."⁵

The people carrying the theory onto AI ask that question about software. Matthijs Maas, a legal scholar then at the University of Copenhagen, argued in 2018 that even narrow AI often runs inside networked, tightly coupled, opaque systems. That makes it "prone to 'normal accident'-type failures" that cascade quickly and are hard to contain or even detect in time.⁶ He drew the bridge to the case on the record, imagining a military "flash war" analogous to the algorithmic flash crashes in finance.⁶ His proposals aim at the levers the theory names: less opacity, varied rather than standardized architectures, a cap on integration to hold coupling down. He would also limit a system's speed and autonomy, and stop treating a human in the loop as a reliable safeguard when the person may be only a "moral crumple zone."⁶ He called his own suggestions "largely speculative."⁶

A 2023 paper by Federico Bianchi, Amanda Cercas Curry, and Dirk Hovy extended Perrow's framework to AI and argued a normal accident is "only a matter of time."⁷ The same authors, though, concede the fact the argument rests on. "Currently, AI models are not tightly coupled: Few systems pipe models together," they write, even as they call the potential high.⁷ They disclaim near-term catastrophe: "we do not assume any fatal risks in AI anytime soon."⁷ Their own examples of AI going wrong are modest: an automated ration denial, a mis-graded exam, a mistranslation. Their prescription mirrors Maas, keeping the coupling low, for instance with a confirmation screen between a language model and a car's controls.⁷

The nearest related worry is algorithmic monoculture, where many decisions lean on the same underlying model and fail together. That is correlated failure from a shared part. Perrow's is a different claim: even parts that each work, and each differ, can interact into a system accident.

None of them can point to what the theory predicts: a deployed AI system whose tightly coupled parts cascaded into catastrophe. The proponents say so. The coupling that would make it possible is, for now, mostly potential.

The systems that run safely, and the measure the theory lacks

The strongest objection came from researchers who went looking for the systems the theory says should fail. In the 1980s a Berkeley group studying High Reliability Organizations examined air traffic control, two nuclear aircraft carriers, and a utility's power system, all complex and tightly coupled by Perrow's description. It found them running for years with almost no catastrophic failure.⁸ In each of the five years before their study, United States air traffic control handled more than 75 million instances of a controller taking an aircraft across an airspace, with no midair collision when both were under positive radar control.⁸ A single carrier's six-month deployment logged over 16,000 arrested landings with no deck accident.⁸

They credited features Perrow's theory says cannot sit together, chiefly an authority structure that hands control to frontline operators at peak danger, then takes it back. If complex, tightly coupled systems can be run this safely for decades, the claim that catastrophe is only a matter of time is in trouble. The Berkeley researchers were careful. They offered it "in the spirit of discovering anomalous data rather than theory disconfirmation." The air-traffic record carries its own bound too. The qualifier "under positive radar control" sets aside the aircraft the system was not watching.⁸

There is a deeper problem. Perrow's two properties have no measure independent of the accidents they explain. When the Berkeley researchers held up air traffic control and carrier flight decks as complex, tightly coupled systems run safely, Perrow reclassified them. He called air traffic control "basically a linear system" and carrier operations "basically linear" and "loosely coupled."² If a system's complexity and coupling can be reassigned after the fact to match whether it failed, then the sentence "AI is complex and tightly coupled, so catastrophe is inevitable" cannot easily be tested. That is the gap where confidence outruns proof, and it opens under the doom and the dismissal alike.