No AI has to go rogue for humans to lose control

The scenario needs no rogue AI, just a lot of reasonable ones

Jan Kulveit, Raymond Douglas, Nora Ammann, Deger Turan, David Krueger, and David Duvenaud made a claim in January 2025: humanity does not need a single AI system to seize control for people to lose it.¹ Their case builds on two earlier arguments. Paul Christiano sketched the mechanism in 2019: machine judgment quietly outcompetes human judgment at whatever gets measured. No single dramatic failure has to cause it: "we ultimately lose any real ability to influence our society's trajectory."² The same year, Andrew Critch and David Krueger made the more general point in a report called ARCHES. A catastrophe need not come from one indecomposable AI system. It can arise from the aggregate behavior of many separate, non-decisive ones.³

Kulveit and his co-authors apply that idea to three places power already runs through: the economy, national governments, and the production of culture. In each, today's system serves human interests, imperfectly but really, because it currently needs human wages, human votes, or human attention. Replace enough of what humans supply with something that works as well, one substitution at a time, and the feedback that kept the system pointed at people fades. Four separate claims carry that argument, each checked on its own terms. AI increasingly captures human attention. It displaces human labor. Institutions come to depend on it rather than on citizens. And these three losses feed each other, until the loop is hard to reverse. None of the four requires a system that wants anything at all.

Curated feeds already control what people see

Recommender systems choosing what appears in a feed are the paper's clearest example of AI already doing culture's job, "not merely as a passive tool but as an unusually active shaper of how humans create and communicate."¹ That much is not in dispute. What's contested is whether curated attention reshapes what people believe and do, the step this argument needs.

The best test available is a large randomized experiment run during the 2020 US election. An independent academic team moved a sample of consenting Facebook and Instagram users off the platforms' normal ranked feed and onto a plain chronological one for three months.⁴ The switch changed what people saw and how long they stayed. More political and untrustworthy content reached them. Less content flagged as uncivil reached Facebook users. More content from friends across the political divide showed up there too.⁴ But on the outcome that matters, whether shifted exposure moved what people believed, the study found nothing. In its own words: "the chronological feed did not significantly alter levels of issue polarization, affective polarization, political knowledge, or other key attitudes during the three-month study period."⁴

AI systems already choose what a large share of people see. That curated attention reshapes belief or votes has not been shown by the best-powered study anyone has run. Three months is not forever. But a case for gradual disempowerment that treats recommender systems as proof AI already reshapes belief is citing more than this evidence supports.

Labor's share of the economy already fell, decades before AI

The paper's case for the economy runs through a plain mechanism: markets serve people today, imperfectly but really, because consumers need money, and they get money mostly by doing useful work.¹ The paper calls that channel stable, unsourced, near 60 percent of GDP "for over a century." The outside record disagrees. US labor's share ran near 64 percent from the postwar years into the mid-1980s, then fell to about 58 percent, and has stayed there through 2023, a real, six-point decline.⁵⁶ That complicates the labor-displacement link more than it confirms it. Both studies trace most of the drop to offshoring and the falling price of capital goods, tied to the broader computer age.⁵⁷ It starts in the early 1980s, decades before any AI system here existed. The one AI-specific channel is task-level. Daron Acemoglu and Pascual Restrepo tracked US commuting zones from 1990 to 2007. One more industrial robot per thousand workers, they found, cut local employment by roughly 0.18 to 0.34 percentage points and wages by 0.25 to 0.5 percent, twenty to thirty times smaller than the six-point aggregate move.⁸ A real version of labor's claim on output shrinking has already happened, mostly for non-AI reasons: neither confirmation of the mechanism nor a refutation.

Even granting a much larger robot-driven effect, the sharpest objection here says the mechanism might not do the work the paper assigns it. Tom Davidson, a researcher with no stake in the paper's authorship, argues human power runs mainly through capital and taxation, not wages. In his words: "Even if AIs earn significant non-subsistence wages, humans can easily tax that income at >50% and give it to humans."⁹ If he's right, an economy where AI does nearly all the work could still leave humans holding the money: owning and taxing the machines doesn't require anyone's labor. Labor displacement is demonstrated, at a modest scale. Economic disempowerment, the loss of power over resources, not just jobs, is a further step nobody has shown yet.

The Netherlands already ran this experiment on itself

The paper argues that states currently need citizens for more than votes: "States need citizens not just as sources of legitimacy through democratic processes, but as willing participants in state functions."¹ The clearest evidence that AI already changes that relationship comes from the Netherlands. Dutch tax authorities used a self-learning algorithm to flag childcare benefit applications as likely fraud. A parliamentary inquiry found the model's risk scoring had used applicants' dual nationality, a practice the country's data protection authority ruled unlawful and discriminatory on 17 July 2020.¹⁰ The inquiry's own verdict was blunt: "basic principles of the rule of law were breached," and thousands of families were treated to "an injustice without precedent."¹⁰ A separate Dutch system, SyRI, matched government databases to flag welfare fraud suspects. On 5 February 2020, The Hague District Court struck it down for being "insufficiently transparent and verifiable," a violation of the right to private life under European human-rights law.¹¹ Both cases are real, documented, and severe: an automated government decision system operated with limited human oversight and caused harm at scale.

Follow both cases to their ending, though, and the evidence complicates the paper's framing. A court struck SyRI down. A parliamentary inquiry exposed the benefits scandal, a cabinet resigned over it, and compensation followed.¹⁰¹¹ The feedback loops this argument fears will go quiet, courts, parliament, public scrutiny, stopped both systems, at real but recoverable cost. The state link's best evidence proves the harm is real. So far, it proves the reversal too.

The step from three real losses to one that doesn't reverse

None of the three links above reaches the ending the paper worries about: that once AI has captured enough attention, displaced enough labor, and been threaded into enough state functions, the three losses stop staying separate. Weaker economic power makes institutions less accountable to ordinary people. Weaker cultural influence removes a channel for organizing a correction. A state that answers less to citizens has less reason to intervene in either.

That last row is the argument's actual disempowerment. Its own authors admit they cannot yet describe it working: "Currently, we lack a compelling positive vision of how highly capable AI systems could be integrated into societal systems."¹ That is argued from first principles, not demonstrated.

Tyler Cowen, an economist with no stake in the paper, doubts its starting line: "When were humans ever in control to begin with?"¹² Markets and democracies, he argues, have always been only imperfectly responsive to the people inside them. His own guess: competing AI systems, and the humans and states building them, will check each other the way market and political actors already do. "'Checks and balances forever' is by far the more likely equilibrium."¹² What he doesn't explain is why the concentration this argument fears, decisions running through no human at all, would stay as bounded as ordinary markets.

Davidson, introduced above, grants the mechanism and doubts the ending. If humans keep more money than AI systems do, he argues, they likely keep cultural and political power too: the paper's three domains are less independent than its story needs.⁹ His sharpest line names who the loop would have to defeat: he is "finding it hard to see, concretely, how the high-level dynamics described here would lead Trump, Xi and lab CEOs to be disempowered."⁹ He doesn't explain why owners and officials who benefit from AI-driven output would keep resisting automating their own oversight, once that got cheaper.

Neither critique retires the worry. What the loop needs and what anyone has shown remain two different things, in both directions.