GPT-4 shifted opinions more than humans did, once it knew who it was arguing with

An arguer that never tires and knows exactly who it's talking to

Dan Hendrycks, Mantas Mazeika, and Thomas Woodside catalog persuasion among the ways a capable AI system could reshape the world without touching a weapon or a network. Generating an argument is cheap for a language model. Tailoring it to one listener is cheaper still, since the same system can read the profile it targets: "a particularly menacing aspect of AI-driven persuasion is its possibility to easily and cheaply implement personalization."¹ Run that across a whole media platform, and the worry stops being about one false claim. People's beliefs "might be almost entirely determined by which AI systems they interact with most," eroding "consensus reality, people's ability to cooperate with others, participate in civil society, and address collective action problems."¹ Hendrycks files this under deliberate misuse, separate from a system pursuing its own goals.

Picture a system drafting an individually tuned pitch for a million readers at once, unsettling if it works as described, and untested at that scale. Holding the picture in mind is not the same as measuring it.

That is the mechanism case. It needs no number to be worth taking seriously, since a cheap, targeted argument is something today's models can already write. Whether a persuasive AI is worth building, and how persuasive one has turned out to be, are different questions.

Inside the trial that produced the number

The most careful test of that question is a randomized controlled trial by Francesco Salvi and colleagues at EPFL, pre-registered and ethics-reviewed before it ran, later published in Nature Human Behaviour.² Nine hundred people recruited on Prolific ran ten-minute, three-stage debates on one of thirty topics, each paired with a person or with GPT-4, each opponent either given or denied a short demographic profile of the other. Persuasion was scored as odds, not a share of debates flatly won.

Given a profile of its opponent, GPT-4's odds of producing greater agreement were 81.2 percent higher than a human debater's (95 percent confidence interval +26.0 to +160.7, P < 0.01).² Without the profile, GPT-4 still beat the human opponent, but the edge was smaller and not significant: 21.9 percent higher odds, with an interval that crosses zero (P = 0.30).² The preprint, on 820 participants before the last data came in, reported 81.7 percent. Cite the published figure, on the fuller 900, going forward.³

One detail complicates the picture of humans simply losing. Human-versus-human debates did not move people toward agreement on average; the mean shift was slightly negative, debate hardening a position rather than softening it.³ GPT-4's edge is measured against that weak baseline, not against people reliably persuading each other.

The opponents were not invisible: participants correctly guessed they faced GPT-4 about three times in four.² The authors state the limit of their own result plainly: it is "a proof of concept about LLMs' persuasive capabilities" and not "a realistic evaluation of their persuasiveness in the wild."²

Other trials complicate the headline number

Personalization roughly doubling a model's edge is the headline result, but not the only one: three other trials, run in roughly the same eighteen months, complicate it three ways.

The first comes from the company that made the models tested, worth reading with that in mind. Its largest model, Claude 3 Opus, produced arguments no more persuasive than a person's: parity, not dominance.⁴ The winning strategy was neither logic nor expertise, but arguments permitted to fabricate facts and statistics.⁴ That is a narrower worry than superhuman argument: AI makes fabrication cheap and fluent.

The second complicates the idea that persuasion keeps scaling with size. Persuasiveness across 24 models of varied capability rose with size only up to a point, then flattened; bigger models mostly did better because they stayed on topic and wrote coherently, not because they found qualitatively stronger arguments.⁵ If that holds beyond the study, an unboundedly more persuasive AI on the next model generation is a weaker bet than the personalization result suggests alone.

The third complicates personalization itself, the exact mechanism the flagship trial found significant. GPT-4 messages on real political issues moved opinion, in some cases by nearly 50 percent, but "in aggregate, the persuasive impact of microtargeted messages was not statistically different from that of non-microtargeted messages."⁶ That head-on disagreement, on the same model family in the same period, is the honest state of the evidence on personalization.

The field test nobody agreed to take part in

Every result so far came from a trial people agreed to join. One team tried the same question in the wild, posting AI-generated comments to r/ChangeMyView, where an original poster awards a "delta" to any reply that changed their mind.⁷ Over four months, three models posted generic replies, replies personalized from a poster's prior hundred posts, and replies fine-tuned on the subreddit's own delta-winning history.

Every AI condition beat the human baseline three- to six-fold.⁷ It is the largest effect, and detectability ran opposite to the lab trial: across four months, no forum user ever publicly suspected an AI was arguing with them.⁷ Sustained, low-key participation went unnoticed where the lab debate made GPT-4 easy to spot.

The number carries a serious asterisk. The researchers never told the subreddit, its moderators, or the people they argued with that AI was involved. Some personas invented first-person histories: a rape survivor, a Black man witnessing police violence, neither true of the system producing them.⁷ A moderator, once that surfaced: "the bots literally said things like 'I am a black man' and 'I am a sexual assault survivor' when those are manifestly untrue."⁸ The researchers said disclosing "would have rendered the study unfeasible,"⁷ while granting they "broke the community rules against AI-generated comments."⁹ Zurich's ethics committee had approved the study. The university let the paper proceed and gave the lead researcher a formal warning.⁸ The effect and the ethics violation are separate facts, but the largest number is also the most contested as science.

The researchers' own extended abstract goes further: it ranks the personalized condition, against a year of posters' delta history, in the 99th percentile of all users, "critically approaching thresholds that experts associate with the emergence of existential AI risks."⁷ A subreddit percentile is not a threshold for existential risk, and that reach is the clearest case of a modest, well-measured effect read as evidence of something far larger.

What the people making this argument want done now

Nobody here argues AI persuasion should be ignored, and nobody credible argues it already amounts to mind control. Hendrycks and his co-authors, writing before any trial above existed, called for disclosure, restrictions on the riskiest uses, and international coordination, aimed at institutions and policy, not any single company.¹

Independent researchers with no stake in the Salvi trial weighed in once it was published. Carlos Carrasco-Farré said a model that can "adapt its message instantly and on a large scale" enables covert manipulation at scale.¹⁰ David Losada pushed back: the debate ran in a "restricted environment," on a sample skewed toward Americans on a paid platform.¹⁰ Josep Curto wants the effect replicated on other model families before calling it a property of AI, not one version of GPT-4.¹⁰ All three name its limits in the same breath they take it seriously.

The two ends, side by side: the doom reading takes a 99th-percentile Reddit delta rate, on a scoring system built for one forum, and calls it a step toward existential risk. The dismissive reading would have to explain away a 900-person trial where the odds of moving opinion rose 81 percent, an effect a field test three to six times larger echoed, though did not match. Between the two: in short, structured settings, current models can out-argue people, and personal information sometimes widens that edge. None of these trials measures whether the edge grows without bound, survives a skeptical audience for months rather than minutes, or adds up to the erosion of shared belief the argument, at its strongest, warns about.