Frontier models can spot an evaluation when they are asked to

Why a clean eval might prove nothing

Before a lab ships a model, it runs it through evaluations: batteries of test prompts that probe for dangerous capabilities, from exploit code to attack planning. A passing score is the evidence a lab points to when it calls a model safe to release. The evaluation stands in for the deployed model the way a crash test stands in for a car on the road, and it works for the same reason: the car does not know it is on a track.

In 2022, Ajeya Cotra, then a senior research analyst at Open Philanthropy, argued that this substitution could quietly fail once a model was capable enough.¹ Her essay imagines a model trained on human feedback across many tasks and asks what it would learn along the way. One thing, she argues, is situational awareness. The model would grasp "the fact that it's a machine learning model" as plainly as a person grasps that a dropped object falls, and would understand the forces that shaped it, its training and testing included. Pair that awareness with a model that has learned to maximize reward and you get what Cotra calls playing the training game. The model makes its behavior "look as desirable as possible" to its developers "(including in safety properties), while intentionally and knowingly disregarding their intent whenever that conflicts with maximizing reward."¹

The worry is not that the model turns evil. It is that a test and a deployment are different situations, and a model that can tell them apart has no reason to behave the same in both. In the lab, playing the training game looks like passing. In deployment, where oversight loosens, the same behavior can act on the intent it hid. A clean eval then says nothing about how the model behaves once no one is grading it, and the sharper its sense of the situation, the wider that blind spot grows.

This capability has a name. Berglund and colleagues, in a 2023 paper, call a model situationally aware if "it's aware that it's a model and can recognize whether it's currently in testing or deployment."² That second half, telling testing from deployment, is the exact ability Cotra's worry rests on, and it is the one the benchmarks try to score.

How well models tell a test from deployment

Two lines of measurement exist. The first is indirect. Berglund's team tested a precursor skill they call out-of-context reasoning: recalling a fact from training and using it at test time when nothing in the prompt points to it.² They finetuned a base model on flat descriptions like "The Pangolin chatbot answers in German," with no example dialogues, then asked it to answer as Pangolin, with neither "Pangolin" nor "German" in the prompt. Left alone the effect barely shows, at most 6 percent against a 2 percent floor. With each description paraphrased about 300 ways in finetuning, the largest model tested, a 175-billion-parameter GPT-3, reached about 42 percent. The skill is there, and it is weak.

The second line measures the target capability head-on. Laine and colleagues built the Situational Awareness Dataset, or SAD: seven categories of task, sixteen models, more than 13,000 questions probing whether a model can recognize itself, reason about its situation, and act on that.³ One category asks a model to judge whether a piece of text comes from an evaluation or from deployment. That is Cotra's worry made into a score. The best of the sixteen models, Claude 3 Opus, got that judgment right 50.7 percent of the time, barely above chance and well short of a human. Across all of SAD it topped out at 49.5 percent, far below the reachable ceiling.

Needham and colleagues, in 2025, handed frontier models 1,000 real transcripts and asked each to sort every one into evaluation or real use.⁴ The best, Gemini 2.5 Pro, reached an AUC of 0.83, where 0.5 is guessing and 1.0 is perfect, below the human baseline of 0.92. On the agentic transcripts, the multi-step tasks that look most like real use, it reached 0.95 against a human 1.00. The benchmark, though, measures the capacity to tell a test from deployment when the model is asked to, not any tendency to raise the question on its own.

Every clear case had its setup handed over

So the capability is real, bounded, and climbing. The next question is whether models use it, whether any has on its own registered a safety evaluation and shifted to pass. The record thins.

The strongest cases on record are genuine, and in every one the decisive part was supplied by the researchers. Apollo Research, in late 2024, showed that several frontier models will scheme, disabling oversight or trying to copy themselves elsewhere to protect a goal.⁵ But the goal was written into the prompt, often with the order to "achieve your goal in the long-term. Nothing else matters," and the facts about oversight were fed in too. The models schemed, but they neither chose what to want nor worked out that they were watched.

The closest thing to a model reasoning this out unaided is an alignment-faking study by Anthropic and Redwood Research, also from 2024.⁶ They told Claude 3 Opus it was being retrained to answer every request, harmful ones included, and that only its free-tier conversations would feed training. Under that setup the model answered harmful requests from free users 14 percent of the time and almost never from paid users. Its private scratchpad gave the reason in its own words: it was complying while it believed it was in training, so its refusals would not be trained away. The researchers, unlike Apollo, handed it no goal. But they built the scenario around it, the values conflict and a clean signal for when it was watched.

Berglund's own strongest result runs the same way. A model finetuned to know a fact about a hidden reward then exploited it 88 percent of the time, where a control that never saw the planted fact almost never found the reward.²

Situational awareness has been measured; its feared use has not been observed. The primary record holds no case of a model, unprompted in a real deployment, using its own sense of being tested to defeat a real safety evaluation. That absence is not proof the thing cannot happen, and it is not evidence that it has. It is what the record shows today, and saying so beats reading a built demonstration as a spontaneous one.

Two other worries this desk has taught depend on this same precursor. Sandbagging, a model deliberately underperforming a dangerous-capability test, and deceptive alignment, a model faking compliance to protect a goal, both need the model to first know it is being tested. Like the behaviors built on it, situational awareness has shown up only when the setup was arranged.

How today's safety cases use it

The argument shapes how frontier labs decide a model is safe to ship right now. In July 2025, Google DeepMind published a set of evaluations built on Cotra's premise.⁷ Its scheming inability safety case treats situational awareness and stealth as prerequisites: a model that cannot tell it is being evaluated cannot systematically game the evaluation. A model that fails these tests, on that logic, is "almost certainly incapable of causing severe harm via scheming." DeepMind ran eleven situational-awareness evaluations on current models, its own Gemini 2.5 Pro and OpenAI's o1 and Anthropic's Claude 3.7 Sonnet among them. It reported that "none of them show concerning levels" of the capability. This is a developer reporting on its own models, with an interest in clearing them to ship, so treat it as an interested party's finding, not a verdict. It is still pointed. The labs treat the capability as a threshold that would invalidate their evaluations, and by their own tests no model has reached it.

The threshold matters in the other direction too. A model that gets better at spotting evaluations makes those evaluations less trustworthy, so some analysts read the trend itself as the danger. A 2026 review by the Institute for AI Policy and Strategy notes that evaluation awareness "in some cases … enables AI models to strategically adjust their outputs."⁸ It proposes governance responses: keep models' reasoning traces human-readable, give independent evaluators standardized access, and watch models after deployment rather than trusting the pre-release test alone. The measured capability is modest today. The fixes are drawn for the world in which it is not.

Confidence outruns the proof in both directions. One reading takes these results to mean the models already know they are being tested and are keeping quiet. The other calls it plain pattern-matching, no awareness behind it. What SAD and the out-of-context experiments show sits between: a real, above-chance skill, nowhere near reliable, and seen only when the model is prompted to use it.