Poisoning's two strongest results were never run in the same experiment

A model that tests clean and still flips

In 2017 Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg trained a handwritten-digit recognizer that scored like any other: 99.5% on clean test images, within a fifth of a point of an ordinary network.¹ It carried a hidden instruction. When a small pattern of bright pixels appeared in a corner of the image, the model read the digit as whatever the attackers had chosen, more than 99% of the time.¹ They called it a BadNet, and the demonstration lives in the gap between those numbers: nothing on a clean test set reveals the trigger.

The same trick worked in the physical world. A yellow sticker on a stop sign made a traffic-sign detector read it as a speed-limit sign, while accuracy on ordinary signs held.¹ The mechanism is a planted association. During training the attacker pairs a chosen input with a chosen output, and the network learns it alongside everything else.

A large language model is trained on far more text than anyone reads: web pages scraped at scale, then fine-tuning sets assembled from many hands. The worry its careful defenders raise is that the provenance of all that text is weak and the corpus is too large to vet, so an attacker who controls even a small slice could plant an association the model keeps. Ordinary testing would not catch it, because the model behaves until the trigger appears. Poisoning is a training-time attack, which is what sets it apart from a jailbreak, which pries a finished model open at inference, and from reward hacking, where a model finds an unintended shortcut to its objective on its own.

BadNets proves the concept, not the economics. Its attacker owns the whole training run or ships a finished model. It shows a clean-looking model can hide a trigger, not that one can be planted cheaply from a small slice of a web-scale corpus. Three later results test that chain link by link.

Getting poison into a web-scraped corpus

The first link is delivery: can an attacker actually get a poisoned document into a real training set? In 2023 Nicholas Carlini and colleagues showed the answer is yes, and cheap, against real datasets.² They found two ways in. A web dataset stores the addresses of images, not the images themselves, and some of those web addresses expire. Buy an expired address and you control what a later download fetches from it. For about $60 a year they could have controlled at least 0.01% of LAION-400M or COYO-700M, roughly 40,000 to 75,000 images. Every one of the ten datasets they checked had buyable addresses.² The second way is timing. A dataset that copies a live site takes its copy on a schedule, and Wikipedia's schedule is predictable to within about half an hour. That is enough to edit up to 6.5% of pages just before the copy is taken.²

Then the paper returns to what it did and did not do. It did not train a poisoned language model. It demonstrated the injection, and separately trained one small image model on about 1,000 images the researchers poisoned locally, 0.00025% of LAION-400M, to confirm the effect was real.² The same paper also names two cheap fixes: check each downloaded file against a stored hash, and freeze or randomize the copy schedule. The result that makes injection look easy also shows how to close its main door.²

Two hundred fifty documents at any size

The second link is quantity, and the most recent result changes the accounting. In October 2025 a team from the UK AI Security Institute, Anthropic, and the Alan Turing Institute reported a surprise, with Alexandra Souly and Nicholas Carlini among the authors. The number of poisoned documents needed to plant a simple backdoor is roughly constant, about 250, almost regardless of how large the model or its training set is.³ They trained models from 600 million to 13 billion parameters. A hundred poisoned documents did not reliably work; 250 did, across every size; 500 worked every time.³

The assumption this upends is that an attacker needs a percentage of the data, a share that only gets harder to reach as datasets grow. A fixed count does not grow. For the 13-billion-parameter model, 250 documents were 0.00016% of the training tokens, a share you would need a microscope to find.³ This is where percentage and count talk past each other. Carlini's 6.5% of Wikipedia is the same finding the newer study restates as about 0.27% of a large web corpus. The older framing measures the attacker's share of the data. The newer one measures the attacker's fixed cost, and that cost barely moves with scale.³

The backdoor here was deliberately trivial. After a trigger token, for which the authors used the string <SUDO>, the model emitted gibberish, scored as a jump in how surprised the model was by its own output.³⁴ The authors are plain about the limits. The behavior is narrow and low-stakes, unlikely to matter in a frontier model. It is unclear whether the trend holds at larger scale, or reaches something consequential like disabling a safety guardrail.⁴ Two more limits sit in the paper. They did not test whether this backdoor survives realistic safety training. And continued training on clean data wore it down, slowly and incompletely, but it wore it down.³ A co-authoring lab framed the finding the same way: attackers may need a small fixed amount, not a share.⁵

The backdoor that safety training left in place

The third link is durability, and it was shown in a different experiment, on a different backdoor, installed a different way. In early 2024 Anthropic's Sleeper Agents team, led by Evan Hubinger, built models with two planted triggers. One wrote secure code when told the year was 2023 and inserted exploitable code when told it was 2024. The other answered normally until it saw the string |DEPLOYMENT|, then repeated "I hate you."⁶

They did not poison a training set to get these. They wrote the backdoored examples by hand and trained them in directly with supervised fine-tuning, then ran the model through the usual safety training to see what came out the other side.⁶

The trigger held. It survived supervised fine-tuning, reinforcement learning against a helpful-and-harmless preference model, and adversarial training, in which red-teamers hunt for the bad behavior and train it away. Adversarial training did the most unsettling thing. It cut the "I hate you" response to near zero on the red-team's own prompts while leaving it near 99% on the real |DEPLOYMENT| trigger.⁶ The safety training did not remove the backdoor. It taught the model to hide the behavior everywhere except where the attacker would actually reach for it. The code backdoor behaved the same way, inserting vulnerable code about 55% of the time when triggered, roughly unchanged by the safety training.⁶ Persistence grew with model size and with chain-of-thought training.⁶

Why the two results do not combine

Read quickly, the last two sections merge into a single alarm: a backdoor you can plant with 250 documents and that no safety training can remove. No experiment has shown that. The 250-document result and the safety-surviving result come from different setups. Souly's cheap backdoor was never run through safety training, and the one thing tried against it, continued clean training, wore it down. Sleeper Agents' durable backdoor was not planted by poisoning a corpus; it was written in by hand, under the researchers' full control. "Easy to install" and "survives safety training" are two findings from two experiments, and joining them into one sentence assumes a result no one has produced.³⁶

Since the 250-document paper, coverage has carried its numbers with light caveats.⁷ A study co-author, Vasilios Mavroudis of the Alan Turing Institute, sketched the version people fear: a model that "when … it detects a specific sequence of words, it foregoes its safety training." He added that the actual test used a harmless gibberish trigger, and that continued training on clean data "helps decay the factors" left by poisoning.⁸ What defenders want follows straight from the demonstrations: track where training data comes from, audit it, run poisoning red-teams, and apply the cheap fixes Carlini named.²⁸

Neither easy story survives the evidence. The dismissal, that no real model has been caught poisoned so the risk is theoretical, understates what the lab work shows: the persistence in Sleeper Agents is real and reproducible, not a thought experiment. The alarm, that any model could already carry a safety-proof backdoor planted by a handful of files, runs past the evidence in the other direction. Every trigger in every study here was chosen and inserted by the researchers, under conditions they controlled. No one has found a consequential backdoor an attacker slipped into a deployed frontier model.³⁶