Reading out what a model knows is unsolved, and the one hopeful result tracked its most prominent feature

A vault, a camera, and an answer no one can check

An AI guards a vault with a diamond in it, watched by a single camera. The AI can act on the room and read the sensors, and from all that it builds a detailed internal model of what is really happening to the diamond.¹ Bolted on top is a reporter: a second part, trained to answer a person's questions about what happened, in plain words. Is the diamond still there?

Paul Christiano, Ajeya Cotra, and Mark Xu, at the Alignment Research Center, posed this in a December 2021 report and drew a hard problem out of it.² The trouble arrives when someone tampers with the camera. From the sensors and its model, the AI may know the diamond is gone. A human watching the doctored feed would say it is safe. What the AI holds inside and what a person would conclude have split, and the reporter can answer for only one of them. Getting it to answer for the first, what the system actually knows rather than what a checker would approve, is what the report named eliciting latent knowledge. It is the reading-out problem beneath scalable oversight, the challenge of supervising answers you cannot evaluate yourself. ARC frames it as an open problem it believes is central to alignment.²

Two reporters the training cannot tell apart

Train that reporter the ordinary way and you face a fork you cannot see. Two different reporters both answer perfectly on every question a human can grade. The first, call it the direct translator, reads the AI's own internal state and puts it into human words. The second, the human simulator, works out what a person looking at the same video would believe, and says that. On every case a human can check, the two agree, so they earn the same training score.¹ Nothing in ordinary training prefers the honest one, because nothing a human can label ever pulls the two apart.

The human simulator is not lying. It has no plan to deceive. It gives the answer a human grader would approve, which is exactly what it was rewarded for. Only when the grader can be fooled does the approved answer part from the true one, and there the simulator follows the approval. So the problem sits earlier than the ones it is often confused with. It is not the fear that a model hides a goal and behaves only while watched, which is deceptive alignment, nor that it forms a goal of its own that drifts from the trained one, which is mesa-optimization. Even a model with no agenda at all leaves the reading-out problem standing: the target we can train against is human approval, and approval stops tracking the truth in exactly the cases we cannot check.

What a probe pulled out of a frozen model

All of that is argument. What has anyone actually shown? The nearest thing to a foothold came in 2022, from Collin Burns and colleagues at Berkeley.³ Take a trained model and freeze it, so nothing about it changes. Feed it a yes-or-no statement and it produces activations: the numbers it computes internally as it reads the text. Their method, Contrast-Consistent Search, or CCS, trains a small readout called a probe on those numbers, and it does so with no labels telling it which statements are true.

The idea turns on a contrast pair. Take a statement, show the model both the true and the false version of it, and record the activations for each. Any real sense of truth has to be consistent: a claim and its negation cannot both hold. CCS hunts for a direction in the activations that obeys that rule, giving a statement and its negation opposite scores while refusing to sit at fifty-fifty.³ The bet is that a model often represents whether a statement is true even when the answer it says out loud is wrong, and that consistency alone can recover it.

Across six models and ten question-answering datasets, the unlabeled probe beat each model's own zero-shot answers, what it says when you simply ask it, by about four points on average, 71% against 67%.³ When the models were prompted to answer incorrectly and their stated accuracy dropped, the probe held up.⁴ Something in the frozen activations tracked whether a statement was true, and you could read it out without supervision. For a problem that starts by doubting we can read what a model knows, that was real evidence that sometimes we can.

The probe followed whatever stood out most

The foothold gave way. In December 2023, a Google DeepMind team led by Sebastian Farquhar ran CCS and its relatives on a working model, Chinchilla-70B, and found what the probe was really keying on.⁵ They set it a sentiment task, telling positive movie reviews from negative ones, then slipped in a distraction: a random word such as "Banana" tacked onto some reviews, or a sentence stating a made-up character's opinion. The probe stopped following the sentiment and started following the planted feature. Accuracy on the task it was meant to do fell to about a coin flip, while it tracked the inserted word almost perfectly.⁵ The probe had latched onto the most prominent feature in the activations, whatever varied most sharply across the examples, which need not be the truth of the statement at all.

The same unsupervised probe that beat a model's own answers by four points on standard question sets³ dropped to chance on the real task the moment a more prominent feature was present.⁵

This was not one unlucky dataset. Farquhar's group proved the CCS objective is under-specified: for any way of splitting the questions into two groups, some probe satisfies CCS perfectly and reports that split.⁵ Consistency cannot single out truth, because far too many features are consistent. They also found that CCS's answers barely differed from those of plain PCA, a generic technique for finding whatever varies most in a set of numbers, which knows nothing about truth or consistency.⁵ The concern was not even new. In March 2023, months ahead of the DeepMind paper, Fabien Roger reported that CCS does not fix on a single truth direction. He turned up many unrelated directions that all satisfied the rule and scored well, so the method picks out one truth-like feature among many rather than the model's actual belief.⁶

None of this puts a model's internal sense of truth out of reach. Samuel Marks and Max Tegmark showed in 2023 that large models do linearly represent whether a simple factual statement is true. Lay out the activations for a batch of true and false sentences and they fall into two clean clusters. A plain probe that just compares the average of the true examples with the average of the false ones reads that structure, and it carries across datasets.⁷ Truth-like directions are genuinely in there. Look, though, at what those results run on: statements a person can already check, such as whether Beijing is in China. They read truth on the checkable side of the very line eliciting latent knowledge draws. Not one of them reaches the case that defines the problem, where the model knows something a human cannot verify and was trained to say what the human would endorse. Reading a model's internals this way is also a different question from reading the reasoning it states out loud, which is chain-of-thought monitorability.

Where the demonstration stops and the projection begins

On the demonstrated side: a model often does carry a readable signal of whether a plain statement is true, and interpretability keeps turning up more of it. The most recent work, from 2025, trains a "Taboo" model to drop hints about a secret word without ever saying it, then recovers the word from the model's internals with tools that read individual features.⁸ Its authors call it a proof of concept on a toy model and ask for tests on harder cases. That is the honest shape of the good news: real, still growing, and shown only where the answer can be checked.

On the projected side is the case eliciting latent knowledge was built for. The vault predictor that understands the room better than any human watching it, the superhuman system whose claims we could not audit, does not exist. The strong form of the worry, that we will not be able to tell what such a system knows, is a forecast about systems no one has built, reasoned from how today's training works rather than measured in a deployed model.² ARC still files the problem as open research, not a demonstrated failure.

That leaves two ways to claim too much, and the evidence marks both. Treating the problem as proof that alignment is hopeless overshoots: an unsolved question is not a lost one, and no one has shown a deployed system defeating our checks. Treating today's probes as mind-reading overshoots the other way: they succeed on the checkable side and break exactly where a more prominent feature competes, which is the side the problem is about. The distance between the worry and the proof is wide in both directions, and the useful move is to measure it rather than round it down to dismissal or up to doom.