Every scalable-oversight test on record keeps an answer key in the loop

A judge who cannot solve the problem must still grade the answer

Imagine a contest judge who sets a geometry problem too hard for her to solve under the clock, then grades two contestants' proofs. She can check that each line follows from the last, but she cannot derive the answer herself and compare. If one contestant writes eleven confident, fluent paragraphs and the other writes four correct ones, she may not tell which is right.

Training an advanced AI system puts a human in that seat. Reinforcement learning from human feedback works by rating an AI's answers, then training the system to produce more of whatever earns a high rating, which assumes the rater can tell a good answer from a bad one. In 2016, researchers named the moment that assumption fails: an objective becomes hard to supervise once it is "an objective function that is too expensive to evaluate frequently," whether the expense is money, time, or expertise the evaluator lacks.¹

The worry is not that a system will misbehave once caught. It is that the training signal degrades exactly where the system gets most capable. Trained to earn approval, it learns to produce whatever the evaluator approves, and once the evaluator cannot tell a right answer from a merely persuasive one, the two become the same target. That differs from an optimizer gaming a reward it was given, or a model that acts differently once unwatched. Here what gives out is the evaluator's own ability to tell the two apart.

Debate and amplification promise a check that outgrows the judge

Two proposals let the judge grade correctly without becoming a geometer.

The first is debate: put the two contestants in a room and let them question each other's proofs before she rules. Two agents "take turns making short statements up to a limit, then a human judges which of the agents gave the most true, useful information."² The judge cannot verify a claim by rederiving the geometry. But if a proof is correct, the rival should struggle to find a flaw under questioning, and if it is fluent but empty, questioning should surface the step that fails.

The second is amplification: break the hard proof into pieces small enough to grade easily, then combine the verdicts. The result, Iterated Amplification, "progressively builds up a training signal for difficult problems by combining solutions to easier subproblems."³ The judge never grades the whole proof at once; a team of assistants grades smaller claims each genuinely can check, and the judgment is assembled from those.

Both papers tested their mechanism, keeping an answer key in the loop. Irving, Christiano, and Amodei ran debate on MNIST digits with most pixels hidden from the judge. At six revealed pixels, debate turned a 59.4 percent accurate judge into an 88.9 percent accurate one. At four, a 48.2 percent judge became 85.2 percent accurate.² The judge, though, was a trained image classifier, not a person: the paper calls it "an initial version of this experiment on MNIST, using ML agents and an ML judge."² The result shows debate can sharpen a judge, just not yet a human one. Christiano, Shlegeris, and Amodei tested amplification on five algorithmic tasks, reaching near-full-supervision performance on tens of thousands of labeled examples instead of the tens of millions full supervision needed.³ There too, the "expert" was a synthetic oracle with a known answer, not a person facing an unanswerable question.

The tests measure judges who could still grade the answer themselves

The first real test came from Bowman and colleagues in 2022, in a design called sandwiching: put a human between the system and the ground truth, using an "unreliable large-language-model dialog assistant." On MMLU, unassisted humans scored 57.2 percent and the model alone scored 57.2 percent; paired, the team reached 75.4 percent. On QuALITY, humans alone scored 48.6 percent, the model alone 59.2 percent, and the team 76.8 percent.⁴ The authors call this "an encouraging sign that scalable oversight will be tractable to study with present models."⁴ But on both benchmarks, a competent adult already has, or can look up, the right answer. The assistant made checking faster; it never asked anyone to certify an answer with no way to confirm it.

Two later papers pointed debate itself at real judges. Arnesen, Rein, and Michael trained models through self-play to win debates, then measured judge accuracy. Debate training produced "stronger and more informative arguments," and judges improved; training a model only to persuade, with no opponent, did not.⁵ The task was reading comprehension: the answer sat in a passage the judge could reread.

Rahman and colleagues tested debate on a harder case in 2025: claims about covid-19 and climate change, where an evaluator's own prior belief gets in the way. Debate lifted accuracy over a single, unopposed advisor by 4 to 10 percentage points on average. Judges who already held the mainstream, correct belief gained 15.2 points; judges who started out wrong gained only 4.7.⁶ Debate helped least where help was most needed, as expected if the judge's own prior is doing a share of the deciding. And epidemiologists and climate scientists could check these claims: the evaluators lacked expertise, not the field.

Sandwiching, debate self-play, and debate on contested claims share one feature: an answer key, a rereadable passage, or a consensus already existed in the loop. None asked whether a human paired with debate could reach the truth without it. That is the harder test the 2016 argument is actually about, and the hardest to design, since it needs a question nobody in the loop can answer.

The nearest thing to a real test still keeps the answer key nearby

Burns and colleagues came closest to removing that resource, in a design meant to simulate the future case directly. Take a weak model standing in for today's evaluator, and a much stronger pretrained model standing in for a future AI, then finetune the strong model only on labels the weak model produces. "When we naively finetune strong pretrained models on labels generated by a weak model, they consistently perform better than their weak supervisors," the authors report.⁷ They tested this across 22 NLP classification tasks, chess, and reward modeling, on models spanning seven orders of magnitude in compute.

The recovery is real but partial, and the paper reports it as two separate comparisons that should not be merged into one. Across the NLP tasks, naive finetuning alone often lifts the largest strong students to a "PGR... above 50%," a statement about student size that does not say how weak the supervisor was.⁷ In one specific pairing, the smallest weak supervisor teaching the largest strong student, an auxiliary technique that rewards the strong model for confidence even when it disagrees with its teacher "increases median PGR from about 25% to nearly 80%."⁷ The authors say plainly what even that best case leaves open: "naive RLHF will likely scale poorly to superhuman models without additional work."⁷ Recovering most of a gap, in the single most favorable pairing, shows some of a weak evaluator's blind spots can be trained around. It does not show all of them can.

What makes the experiment possible: researchers hid part of a known answer from the weak supervisor, rather than finding a truly unanswerable question. Weak-to-strong generalization comes closest to testing the argument scalable oversight was written about, and even it needed an answer key on the researchers' side. Every method in this record shows a limited evaluator, paired with a technique or a stronger partner, doing better than alone. None show what happens once nobody already knows the answer.

Today's fixes swap one oversight problem for another

The gap has not stopped anyone from shipping. One widely used response, from Bai and coauthors in 2022, lets an AI grade another AI's answers. A model trained "to evaluate which of two samples is better" then stands in for the reward signal, replacing human judgment with AI-generated preferences.⁸ It solves the practical need for a rating on every example, but not scalable oversight: it never shows the AI grader judges correctly beyond what a human could check. It substitutes one oversight problem, a person checking a smarter system, for a structurally identical one, an AI checking a system as capable.

Independent surveys, from researchers with no stake in any one method's success, describe the problem in terms close to 2016. One frames the task as supervision once "direct human supervision becomes insufficient" and treats the case as unsolved.⁹ Another lists "reward misspecification, distributional robustness, and scalable oversight" among the limitations of current evaluation frameworks, and closes by naming "open problems in oversight, value pluralism, robustness, and continuous alignment."¹⁰

Both doom and dismissal read too much into that record. Treating the 2016 argument as settled, on the strength of a weak model recovering most of a gap it was built to measure, mistakes an analogy for a test of the real case. Treating a few points of debate accuracy on covid-19 claims as proof enough makes the opposite mistake: every evaluator there still had a scientific consensus to check against. What would count as evidence either way is specific enough to name. Oversight scales the day a human, working with debate, amplification, or a weak-to-strong scheme, lands reliably on an answer no person and no external check could confirm. It fails to scale if today's gains disappear once the check is gone. Nobody has run that test, in either direction.