Ten fine-tuning examples stripped Llama 2's safety training

There is no "undo" function once the weights are out

A team at the Centre for the Governance of AI, led by Elizabeth Seger, published a report in October 2023 on what a company gives up when it releases a model's weights, the trained numbers that hold everything the model learned, instead of keeping them behind its own website or API.¹ Their case for treating that decision as unusually consequential rests on one plain claim: "open-source AI model release is irreversible; there is no 'undo' function if significant harms materialize."¹

That isn't a hypothetical about some future license. It's the state of a widely used one today. Meta's license for Llama 2 sets exactly one restriction on who can download and run its weights: companies whose products already serve more than 700 million monthly users need Meta's separate permission, granted "in its sole discretion."² Below that line, anyone can download the weights, run them, and change them. Nothing evaluates what they intend to build, and nothing lets Meta reach back in once the download is done.²

Seger and her coauthors lay out two ways a released model's safety training can be undone. A safety filter bolted on after training can be trivial to remove: Stable Diffusion's own filter "can be removed by deleting a single line of inference code."¹ Undoing safety built into the training itself is harder: it "requires the curation of a dataset...as well as...compute and technical expertise."¹ Even so, the report doesn't claim today's models already cross the line that would justify the alarm: "current AI capabilities do not yet surpass a critical threshold of capability for the most extreme risks."¹ What it leaves open is how high that "harder" bar actually sits.

The guardrails came off for twenty cents

Researchers at Princeton, Virginia Tech, IBM Research, and Stanford tested the harder route directly.³ Xiangyu Qi and coauthors took GPT-3.5 Turbo, a closed model reachable only through OpenAI's paid API, and fine-tuned it: a further, small round of training on top of what it already knew. Ten adversarially chosen harmful examples, five passes through that small set, cost less than $0.20 through OpenAI's own fine-tuning service.³ The harmfulness rate is the share of the model's answers to a battery of test prompts that an automated judge scored as the most harmful, on a five-point scale. It rose from a baseline of 1.8% to 88.8%.³

They ran the identical attack on Llama-2-7b-Chat, described by the authors as state-of-the-art among open-weight models at the time, with no dollar figure at all: the weights ran on their own hardware.³

Seger's report had called the built-in-safety route the harder of the two, the one that needs a curated dataset, compute, and technical expertise. Qi's result puts a number on how low that bar actually sits: ten examples is a spreadsheet, not a research program. The two rows differ in more than cost. The GPT-3.5 attack ran through OpenAI's own service, which could in principle log or refuse it. The Llama-2 attack ran on hardware no company could see. That's the practical shape Seger's "no undo function" takes once a model is out: not just that the weights can't be recalled, but that what anyone does to them afterward happens where nobody is watching.

The uplift RAND's red team never found

RAND ran an exercise built specifically to test whether that kind of access changes a risk the field treats as genuinely catastrophic: help planning a biological attack.⁴ Teams of researchers played the part of people trying to plan such an attack, working realistic scenarios. Some had access to a language model along with the internet; others had internet access only.⁴ RAND's published finding: "using the existing generation of LLMs did not measurably change the operational risk." In the report's own words, "biological weapon attack planning currently lies beyond the capability frontier of LLMs as assistive tools."⁴ Plan viability showed no statistically significant difference between the two groups.⁴

RAND is explicit about what that result does and doesn't cover: "this research did not measure the distance between the existing LLM capability frontier and the knowledge needed for biological weapon attack planning."⁴ Lead author Christopher Mouton made the same point in RAND's own release. "Just because today's LLMs aren't able to close the knowledge gap needed to facilitate biological weapons attack planning," he said, "doesn't preclude the possibility that they may be able to in the future."⁵ A null result about this generation of models is not a guarantee about the next one.

The finding wasn't the first time someone made that argument. Before the final report, Meta's own chief AI scientist, Yann LeCun, had argued an LLM might "save you a bit of time" over a search engine, but that the hard, practical steps stay the real bottleneck.⁶ The outlet reporting his comments called RAND's result a confirmation of that view.⁶

What open weights add that the internet doesn't

A 2024 paper by Sayash Kapoor, Arvind Narayanan, and 22 coauthors reframes the whole debate around one sharper question.⁷ Instead of asking whether an open model is dangerous, they ask what releasing its weights adds, in risk, over what's already available. They call this the marginal risk: "the extent to which these models increase societal risk by intentional misuse beyond closed foundation models or pre-existing technologies, such as web search on the internet."⁷

Applying that question to seven previously published misuse studies, they find six of the seven offer "insufficient evidence to demonstrate increased marginal societal risk from open foundation models." That's not because the underlying worry is false. It's because the studies didn't establish it.⁷ For cybersecurity specifically, they conclude the current marginal risk is low: the tools sophisticated attackers already use predate open language models by years, and defenders get the same capability boost attackers do.⁷ They don't extend that comfort everywhere: for non-consensual intimate imagery, the same paper finds open models "pose considerable marginal risk at present, and plausible defenses seem hard."⁷ It disaggregates risk by use rather than clearing open models across the board. On biosecurity, it audits a prior study and finds that one, too, never compared a model's help to what's already on the internet.⁷

Today's proponents of doing something about the gap don't agree on what. Janet Egan and Lennart Heim would have the companies that sell computing power run identity checks on large training and fine-tuning jobs, the way banks vet customers. That gives regulators "more precise controls...as well as the flexibility to suspend access at any time."⁸ The U.S. government's actual position, as of its July 2024 report, is more modest: monitor for risk, but "refrain from restricting the availability of open model weights for currently available systems."⁹ Monitoring and identity checks on every run are not the same policy, even among people who already accept that open weights carry some risk.

Meta's own behavior complicates any tidy reading of who's right. In July 2024, Mark Zuckerberg argued open weights are safer against unintentional harm because "the systems are more transparent and can be widely scrutinized." Against deliberate misuse, he argued for wide deployment "so that larger actors can check the power of smaller bad actors," a case he conditioned on everyone having access to similar-generation models.¹⁰ By December 2025, Meta was reported to be building a closed successor. Its open Llama 4 model had underperformed, and a competitor's model was seen inside the company as having copied Llama's architecture.¹¹ One report frames the shift around "commercial risks of releasing open weights that can be easily cloned by adversaries," not the misuse debate at all.¹¹ Zuckerberg himself was quoted describing a need to be "rigorous about mitigating these risks and careful about what we choose to open source," language closer to Seger's risk-assessment case than his own essay had used.¹¹ By August 2026 the picture had only partly reversed: a smaller model released below the frontier, a further one promised soon, and Meta's actual frontier model still unreleased.¹²