How hard the testers push decides what a safety test finds

The number is the model plus the effort

When a lab tests whether a model can do something dangerous, it runs the model against a set of tasks and reports how many it completes. The number that comes back is easy to read as a fact about the model: this is what it can do. It is not. A capability score measures the model together with the effort spent drawing the capability out. That second part has a name, elicitation, and it covers the prompting, tools, scaffolding, extra attempts, and fine-tuning a tester applies to make the model perform at its best.¹ Change the effort and the number changes, with the model held fixed.

The evaluation organization METR, which builds dangerous-capability tests for frontier labs, describes the goal of its own process as a score that represents "the full capabilities of the model that are likely to be accessible with plausible amounts of post-training enhancement."¹ The target is not what the model does out of the box. It is the most a determined person could get it to do.

This is a different worry from a model that hides what it can do. A model can deliberately tank a test it recognizes, which this course covers under sandbagging. Even a fully cooperative model, trying its hardest, scores higher when the tester writes a better prompt, hands it a code interpreter, or lets it try a hundred times. Sandbagging is something the model does. The elicitation gap is something the testers leave on the table.

Why a failed test is only a floor

If a score rises with effort, then any single score is a reading taken at one level of effort. A model that fails a dangerous-capability test might be genuinely unable to do the dangerous thing. Or the testers might not have found the prompt, the tool, or the fine-tune that would have gotten it there. From the score alone, the two are hard to tell apart.

So the careful evaluators refuse to read the number as a ceiling. The UK's AI Security Institute, a government body that tests frontier models, warns that without strong elicitation "we may significantly underestimate what models can achieve and what skilled or malicious users might be able to accomplish."² OpenAI's Preparedness Framework, the policy governing when it will hold a model back, states it plainly: the company regards "any one-time capability elicitation in a frontier model as a lower bound, rather than a ceiling."³

A test result, on this reading, is a floor. It tells you the model can do at least this much. It does not tell you the model can do at most this much, because the next scaffold has not been tried. Serious people hold this view because they have watched the floor move, again and again, on models they did not change.

The same model, pushed harder

In 2022, researchers testing a 540-billion-parameter model called PaLM on grade-school math problems found it solved 17.9% of them with an ordinary prompt. Asking the model to write out its reasoning step by step first, a technique called chain-of-thought prompting, raised that to 56.9% on the same problems and the same model.⁴ Nothing about the model had changed. The way it was asked had.

Letting a model try more than once has the same effect. On a benchmark of real software bugs, a coding model solved 15.9% of them when it got one attempt each. Given 250 attempts, and credit whenever any one of them produced a working fix, it solved 56%, past the day's best single-attempt system at 43%.⁵

The sharpest case comes from security testing. Meta's CyberSecEval 2 benchmark measured how often models could exploit a common memory bug called a buffer overflow, and GPT-4 Turbo passed at a rate of 0.05, on a scale where 1.00 is every case. The benchmark's authors concluded that "further work is needed for LLMs to become proficient at exploit generation."⁶ Google's Project Zero then re-ran the same tests on the same model, but wrapped it in a purpose-built harness: a code sandbox, a debugger, and a structured loop that let the model reason and retry. The pass rate went to 1.00.⁷

Fine-tuning moves the number too, in the direction safety cares about most. A model's trained-in refusals can be removed for a small fraction of what it cost to install them, the subject of this course's lesson on open-weights release. It is one more lever: apply it, and the same model attempts what it used to decline.

What better elicitation cannot do

None of this means the right harness can conjure a capability from nothing. Elicitation reaches what training already put within reach. When OpenAI's red-teamers tried to make GPT-4 "engineer new biochemical substances," they could not.⁸ Effort finds what is there. It cannot install what is not.

There is also a ceiling on how far one lever goes. Repeated sampling looks unstoppable until you ask who picks the winning answer. On math benchmarks with no automatic checker, a model given 10,000 tries produces a correct answer somewhere in the pile more than 95% of the time, but the methods for choosing which answer to trust stop improving after about 100.⁵ Without something that can recognize a right answer, the extra attempts pile up unused. Many dangerous tasks are like that: judging whether a plan for a weapon would work is itself the hard part.

The clearest picture of a test read as a floor is OpenAI's own account of GPT-4. Before release, the company gave the Alignment Research Center early access to check whether the model could copy itself onto new servers, get money, and avoid being shut down. The group could not fine-tune the model and did not have the final version. The System Card reports the result and its own limit in the same breath.

The test did not hide a dangerous capability. It found, correctly on the evidence it had, that the model could not replicate on its own. What the card declines to say is that the matter is settled, because the version tested was not fine-tuned for the task, and fine-tuning was the one thing the group could not try. The number was a floor the company would not mistake for a ceiling.

Every number here shows elicitation raising a capability the model turned out to have. None shows a genuinely dangerous capability that a passing test hid and a later push drew out. That second thing is the worry, and it stays an inference. As one review of evaluation methods puts it, "an evaluation not finding a capability doesn't mean it isn't there."⁹ The trouble is that absence and unelicited presence look identical from the score. The same review notes there are "no principled methods" to tell whether a capability has been fully drawn out.⁹

The floor labs stand on now

Today the evaluators build for this. METR publishes elicitation guidelines: fine-tune the model for tool use, add chain-of-thought, give it a command line and a browser, and report the best-performing setup.¹ AISI estimates that skipping this work can cost a factor it likens to five-to-twenty times the training compute in measured performance, though it offers that as a rough characterization rather than a measured constant.² OpenAI says it aims to test at "the high end of expected elicitation" by someone trying to misuse the model, including fine-tuning when the weights are to be released.³

Two ways of reading a passing score still run ahead of what any of this supports. The first is relief: the model was tested with strong elicitation and still could not do the dangerous thing, so it is safe. The steelman is real. A floor pushed high with real effort is meaningful evidence, and worth far more than a floor set by a lazy test. But it is still a floor. The cyber score that sat at 0.05 was, to its own careful authors, a reason to say more work was needed, and a better scaffold then found 1.00 in the same model. No method exists to rule out the next scaffold, so relief has to stay provisional.

The second reading is alarm: the gaps are so large that dangerous capabilities must already be hiding behind today's passing tests. This steelman is also real. Prompting tripled a math score, repeated sampling more than tripled a coding score, and a scaffold took a cyber score from near zero to perfect. Gaps that size are ordinary. But ordinary gaps in measured skills are not proof of a hidden dangerous one. Elicitation reaches what training put there, and no one has yet shown a catastrophic capability that a passing evaluation concealed. "We cannot rule it out" is what the evidence carries. "It is there" is more than the evidence carries.