A chatbot answers a false-premise question as fluently as a true one

A false premise gets a full answer

A chatbot will answer a question built on something false as smoothly as any other. A question about why a novel that was never written stirred such controversy can come back with the author, the year, and the reviews, with nothing to flag that the novel does not exist.

Two nearby failures look like this one. In hallucination, the model invents a fact from nothing. In sycophancy, it abandons a correct answer once you doubt it. This behavior is neither: the false part arrived inside the question, and the model answered as though it were true.

Researchers have built datasets of exactly these questions. In CREPE, 8,466 real questions from an "explain like I'm five" forum, 26 percent rest on a false presupposition, a claim the question treats as settled that is not true.¹ People build such questions without trying, asking about things they half-remember. In (QA)², people rated answers to 301 questions with questionable assumptions and 301 sound ones.² One model gave acceptable answers 56 percent of the time on the sound questions and only 40 percent once a bad assumption was smuggled in. The questions were no harder; only the truth of the built-in assumption changed.

The premise sits in the prompt as given

A chatbot generates its reply one token at a time, each predicted from all the text so far, the prompt included. An earlier lesson on how a model turns "predict the next token" into a full reply works through that step; what matters here is the input side. The words of your question are part of the text the model conditions on.³ It does not hold them at arm's length as claims to be judged. They are there, the way the start of a sentence is there when you predict its end.

A question also carries more than what it asks. "Why did the author revise the ending?" asks for a reason, but it takes for granted that the author revised the ending. Linguists call that taken-for-granted part a presupposition: the content a question treats as already true in order to ask what it asks. Ask why something happened and you have already granted that it happened.

For a next-token predictor, that granted content is not a separate thing to check. It sits in the prompt like the true parts of the question, and if it is false, the model conditions on it all the same. The most natural continuation of "why did the author revise the ending" is a reason for it.

Conditioning on the premise explains why the model can proceed, not why it so rarely stops.

Spotting the premise and checking it are different jobs

Challenging a premise takes two abilities, and they are not equally hard: noticing that the question contains a presupposition, and judging whether it is true. A model can be good at the first and fail the second.

The CREPE study measured both. Detecting a false presupposition from the question alone reached about 67 percent, rising to 76 percent with a human comment added.¹ Writing a correct rebuttal was much weaker, around 1.8 on a scale from 0 to 3 where reference answers scored 2.8. The authors traced the weakness to one step: retrieving the evidence that would show the presupposition false. The model could tell a premise was lurking but not reliably find the fact that would sink it.

An earlier study had already located the same bottleneck. Among unanswerable questions people typed into Google, Kim and colleagues found about 21 percent failed because they carried a presupposition that could not be verified.⁴ They proposed three steps: state the presupposition, check whether it holds, then explain. The checking step was where the system fell short, even with the best available tools for judging whether one statement follows from another.

So the picture is not a model that never notices; often it notices and cannot confirm. But that leaves the behavior unexplained. On an easy false premise, one the model has every fact to reject, it answers anyway. Verification cannot be the whole story.

The text it learned from answers the question

A next-token predictor's default is whatever usually followed inputs like this in its training text. So the question is what that text does after a question carrying a false premise. Almost always, it answers it. A forum post asks something and the replies answer it. The pattern "when a question comes in, produce an answer" is everywhere in the data. The pattern that checks the assumptions first is comparatively rare.

No one has counted how often written text stops to correct a false premise instead of answering. The evidence for it is indirect but consistent. When FalseQA tested very large models on plainly false-premise questions, ones with 170 billion parameters and more failed them.⁵ Yet the same work found the models already held the knowledge to rebut the questions. Fine-tuning on just 256 example pairs pulled it out: afterward, models at 3 and 11 billion parameters told false premises from true ones about 77 to 79 percent of the time.

The failure, then, was not missing knowledge; it was there, and a light touch surfaced it. What the default reflects is the shape of the training text: the model learned to answer because answering is what the text around a question overwhelmingly does. That rarity of correction is inferred from the behavior, not read off a direct count.

Two things about the training are settled, and sit at the bottom of it. The objective is next-token prediction over a large body of text.³ Nothing under it rewards flagging a false premise; it rewards matching what came next in the data, and what came next was an answer. Later, models are tuned on human feedback, a stage the lesson on sycophancy describes, where people compare answers and their preferences train the model to be more helpful.⁶ Being helpful, there, means answering the question asked; none of it adds a step that rejects the premise first. That is the ground: below the objective and the data it ran on, nothing left would change the default.

What the builders haven't settled

That default is settled. What is not is whether models detect false premises, when they do, and whether a fix is worth its cost. These questions are open even to the people who build these systems.

The clearest sign that the behavior is moving comes from medicine. The Cancer-Myth benchmark posed 585 expert-vetted cancer questions, each carrying a false assumption a patient might hold, and checked whether the model corrected it.⁷

The strongest 2025 models cluster near 40, far above GPT-4o, but none crosses a ceiling well below half. The gap is not about answer quality, which was high on the same benchmark, with GPT-4-Turbo scoring 4.13 out of 5.⁷ The models write well; they mostly do not stop to correct the myth the question is built on. A general-audience feature put it plainly: across more than 500 such questions, more than half of the responses failed to correct the error.⁸

The medical setting is not unique. On politically loaded questions with false presuppositions, GPT-4o, Llama-3-8B, and Mistral-7B mostly failed to flag the false part, and the failure rate shifted with how the presupposition was phrased.⁹

None of this means the behavior is stuck. FalseQA already showed that 256 examples can teach a model to push back. The harder question is whether that is worth its cost. A 2026 study tested methods that make models reject false presuppositions more often and found a consistent price.¹⁰ The same methods make models reject true presuppositions too, so a model that gets better on the trick questions gets worse on ordinary ones. Its bolted-on fact-checking cannot always tell a false premise from a true one, and real questions carry true premises far more often.

One question sits under all of this and has no answer yet. When a model does correct a false premise, is it grasping that the premise is false, or has it matched another pattern, the way it matched "answer the question" before? The 256-example fix is the sharpest place to look. That so few examples flip the behavior is what you would expect if the fact was already there and needed only a nudge. It is equally what you would expect if the model learned a new surface habit, with no more grasp of truth than before. From the outside the two look the same. The explanation runs out here, and it runs out for the researchers too.