Tell a model a puzzle isn't Tower of Hanoi and it solves Tower of Hanoi anyway

Told outright it isn't Tower of Hanoi, the model solves Tower of Hanoi

Jang and colleagues built a test to diagnose this failure: a puzzle whose prompt states, in so many words, this is NOT a Tower of Hanoi problem. Two reasoning models tested against it, Qwen3-32B and OpenAI's o3, proceed to answer it as a standard Tower of Hanoi problem anyway, as though the sentence denying that had never been written.¹

It is not a one-off. Researchers at Arizona State built a benchmark to catch this pattern at scale. They took 25 well-known logic puzzles, the kind shared as chatbot "gotchas." They turned them into 149 variants that keep each puzzle's reasoning structure intact but change the detail that decides the answer.² Across eleven models, systems that come close to solving the original, unaltered puzzles every time drop sharply on the altered ones. Most of the failures are exactly this behavior: the model produces the puzzle's old, canonical answer instead of working out the new one. The paper's own name for it is "phantom recall."²

A second benchmark, built independently around a family-relationship riddle rather than logic puzzles, put a sharper number on the same collapse. Nezhurina and colleagues fix a template, "Alice has N brothers and she also has M sisters. How many sisters does Alice's brother have?", and vary only the numbers. Most tested models score at or near zero. Among the few that clear even a low bar, GPT-4o answers correctly about two times in three (p=0.649) and Claude 3 Opus a little under half the time (p=0.431).³ On a harder version of the same template, GPT-4, GPT-4o, and Claude 3 Opus all fall below one correct answer in five. OpenAI's o1-preview holds close to perfect across every variation. But o1-mini, sharing that same reasoning lineage, collapses right alongside the older models.³

The old answer already has the higher probability

A model does not check a puzzle against a stored answer key. It predicts one token at a time, a probability across its whole vocabulary for what comes next given everything written so far. Training sets those probabilities, and it raises the odds of whatever continuation followed similar wording most often. A famous puzzle's canonical wording, worked through in exactly its canonical way, appears in training data many times over. It sits in riddle collections, in explainers about the trick, in forum threads that repost the question next to the classic answer, and in the benchmarks built to test whether models still know it. None of that repetition goes away when someone changes one number or one clause in the copy they type into a chatbot. The rest of that sentence still looks, token for token, almost exactly like a version the model has already seen paired with a very high probability, well before it reaches the one word that changed.

How much repetition matters is measurable outside puzzles altogether. Razeghi and colleagues compared GPT-family models against how often a task's key terms appeared in the training corpus. On tasks of matched difficulty, accuracy on the most frequent tenth of terms can exceed accuracy on the least frequent tenth by more than 70 percentage points, the gap between a task a model gets right almost every time and one it gets right hardly at all.⁴ McCoy and colleagues confirmed the same lever on tasks with exactly one correct answer, so difficulty cannot be the excuse. Reversing a sequence of words is the identical mechanical step either way, yet GPT-4 hit 97% accuracy when the correct output happened to be an ordinary, high-probability English sentence. It scored only 53% when the correct output was a low-probability one.⁵ How common the answer sounds can matter more than how hard the task is.

That frequency-accuracy link also supports a claim no puzzle study has tested directly. The override should run strongest on the puzzles repeated most in training, a Monty Hall or a surgeon riddle, and weakest on an obscure one nobody has typed into a chatbot before. No study measures reversion rate against a puzzle's fame. The claim is a reasonable extension of Razeghi's and McCoy's numbers, not a separately measured result.

The model can restate the edit and still ignore it

Jang's team built a second version of the same test that rules out the obvious excuse, that the model simply missed the changed word.¹ Take the textbook Fibonacci rabbit-population puzzle, where a founding pair produces a new pair every month and the population grows the familiar sequence, and rewrite the setup so the founding pair is described as permanently infertile. Read straight, that single change means the population never grows past two. A version of the model without extended reasoning gets this right and says the population stays constant.

That is what separates this failure from three others this desk has already covered. Irrelevant context leaves the original problem intact and adds a true but useless sentence on top. Here the problem's real answer has changed, and the model overwrites the change instead of the added sentence. Memorization is a model reciting text it saw verbatim in training. Nothing in Jang's altered prompt ever appeared verbatim anywhere, so there is nothing to recite: the wrong answer is generated fresh, from a template, not looked up. And a prompt-sensitivity failure turns on how a question happens to be phrased. This one does not. The model in Jang's test could accurately quote back the very sentence it then discarded.

The same override shows up outside riddles, too, wherever a model has a strong default and a prompt asks it to use something else. Wu and colleagues asked GPT-3.5 to write a drawing program under a coordinate convention flipped from its training default. The share of its outputs that even parse as a valid program falls from 99% under the ordinary convention to between 62% and 75%, depending on which convention the prompt asked for instead. The model behaves as though the old convention still held.⁶

More reasoning training does not reliably beat the override

On one point, the sources above are not divided. A model completes the pattern it has seen assigned the highest probability, and a strong enough prior can dominate a prompt that changes only a small, specific part of that pattern. Every benchmark cited here measures some version of that, on different puzzles, different models, different years.

What it proves about "reasoning" is not settled the same way. One reading holds that it shows there is no reasoning happening in any operative sense, not a performance gap that more scale would close. Konstantine Arkoudas argues exactly this from his own testing of GPT-4 on puzzles he designed himself. In one, the model states a relevant fact correctly and then fails to apply that same fact one step later. Arkoudas concludes that "GPT-4 at present is utterly incapable of reasoning, in spite of its sporadic displays of ingenuity."⁷ A milder reading says only that the prior wins under conditions like these, without settling what sits underneath it. Nezhurina and colleagues, whose own numbers are among the sharpest evidence for the override, decline the stronger claim outright: "We do not conclude entire absence of reasoning from the observations."³

Whether training a model to reason longer fixes the override is even less settled, and depends on which model and which puzzle gets picked. PHANTOM RECALL's own extended "Thinking" mode helps substantially on the identical 149-puzzle set: o3 gains 30.2 points over its non-thinking mode, Gemini 2.5 Pro gains 10.1, and Claude gains 8.0.² But o3's thinking-mode score is still only 59.73%, and the paper's own authors call the improvement "narrow and model-specific," concluding that phantom recall "persists across all systems."² The AIW numbers point two directions inside a single family. O1-preview holds near-perfect across every version of its hardest puzzle. O1-mini, trained from the same reasoning lineage, collapses on the identical task.³ Jang's rabbit and Tower-of-Hanoi examples go further than either. There, the reasoning-trained model overrides the stated condition where its own non-reasoning counterpart does not: extended reasoning makes this specific failure worse, not better.¹ An independent replication of the AIW method across the wider family of reasoning models reaches a similarly mixed verdict. Some models are far more robust than others, and the split does not line up with the benchmarks used to market them.⁸

No source here supports a model family having solved the override, or reasoning training never helping. The specific puzzle and model predict the outcome, not a rule running cleanly either way.