A big part of Apple's "reasoning collapse" was models hitting an output limit

What Apple put on the stand

A reasoning model is an ordinary language model trained to write out a long private chain of steps before it commits to an answer. The team behind DeepSeek-R1, one of the models on trial here, showed that this behavior can be produced by reinforcement learning alone, "obviating the need for human-labeled reasoning trajectories."¹ The same training draws out patterns like "self-reflection, verification, and dynamic strategy adaptation."¹ OpenAI's o1 and o3, Claude with extended thinking, and DeepSeek-R1 are the systems people mean by the term.

In June 2025 six Apple researchers put such models through a test built to dodge the usual complaint that benchmark questions leak into training data.² Instead of math problems, they used four puzzles, each with a knob for difficulty. Tower of Hanoi moves a stack of disks one at a time. Checker Jumping and River Crossing slide pieces and passengers past each other. Blocks World restacks blocks into a target tower. Turning the knob adds disks, checkers, people, or blocks. For every run they recorded two numbers: whether the final answer was correct, and how long the model's chain of thinking ran.²

The two results everyone quotes

The finding that traveled is in the first line of the abstract: past a certain difficulty, "LRMs face a complete accuracy collapse."² Below the threshold the models are fine. Above it they score essentially zero, and piling on more of the same puzzle does not bring them back. Apple sorted the behavior into three regimes: on easy puzzles a plain model with no thinking step does better; in the middle the thinking earns its keep; and at the top "both models experience complete collapse."²

The second result is stranger, and it did more of the rhetorical work. As the puzzles got harder the models first spent more thinking tokens, then past the threshold spent fewer, "despite having remaining token budget."² A model that meets a harder problem and thinks less looks like a mind giving up, and that is the reading that spread. It is also the reading that outruns the evidence, because "spent fewer tokens" and "gave up" are different claims. An engineer who worked through the traces put the limit of the finding plainly. All the data really shows is that the models "don't want to" keep going.³ That tells us how the model behaved. It does not measure a ceiling on what the model can do.

Tower of Hanoi runs longer than the model can write

Tower of Hanoi has an exact price. Moving a stack of N disks always takes 2^N − 1 moves, with no shortcut available.⁴ Ten disks is 1,023 moves; fifteen disks is 32,767; twenty disks is over a million. A model solving the puzzle in the format Apple required has to print every one of those moves in order. The rebuttal, "The Illusion of the Illusion of Thinking," did the arithmetic the automated grading skipped. At roughly ten tokens per move, a fifteen-disk answer runs to about 327,670 tokens.⁴ That is several times the output any model in the study could produce in one go.

The table marks where the wall falls. Apple capped thinking at 64,000 tokens, and a little past twelve disks the correct move list no longer fits in that space, whatever the model knows.² The models were not silent about the squeeze. The rebuttal collected traces where a model lays out the pattern and then stops on purpose, saying so in as many words.⁴

Apple's grader counted those cut-off runs the same as a wrong answer.⁴ The clearest sign that this is a mistake is what happened when the rebuttal took the wall away. Asked to hand back a short program that generates the moves rather than the moves themselves, the same models solved Tower of Hanoi at fifteen disks "in under 5,000 tokens," a size the original paper had logged as total failure.⁴ The model knew the moves. It ran out of room to write them down.

Some of the graded puzzles had no answer

River Crossing exposed a sharper problem. The puzzle asks you to ferry people across a river in a boat that holds only so many, never leaving a forbidden group alone on either bank. For a boat that holds three, the puzzle has no solution at all once you pass five pairs of people. It is not merely hard, it is impossible, and provably so with pencil and paper.⁴ Apple's benchmark included those impossible instances and scored the models as failing them.⁴ A model that correctly saw there was no answer earned the same zero as one that flailed. Part of the River Crossing "collapse" is the grader marking models wrong for not doing the undoable.

This does not clear the models. Apple also reported that they often failed after only a handful of moves on crossings that do have answers, well short of any length limit,² and the contrast with their long Tower of Hanoi runs is real. The unsolvable instances inflate the collapse. They do not invent it.

What the puzzles still show

Take out the token wall and the impossible puzzles and something the internet's version of the paper skips over is still standing. The effort-drop result does not depend on either flaw: the models spend fewer thinking tokens on harder problems with thinking budget still to spare, so a shortage of room cannot be the whole story.² Whatever that is, it is a real and odd property of how these models decide when to stop working.

The paper's narrower claim also holds: the models "fail to use explicit algorithms."² Apple wrote the exact recipe for Tower of Hanoi into the prompt and still watched accuracy fall apart at about the same point.² The token wall clouds even this test, since running the recipe still means printing every move, but the result fits a pattern Apple's own lab had found a year earlier. Their GSM-Symbolic study showed that changing only the numbers in a grade-school math problem lowers accuracy, and that adding one irrelevant sentence drops it by "up to 65%" across every leading model.⁵ These systems run long procedures unreliably and lean on the shape of problems they have seen. That was true before the puzzles and it is true after.

What the puzzles do not carry is the sentence they get cited for. "Reasoning models cannot reason" is a claim about capability; a model that could name the millionth move but cannot type it has hit a limit of format, not of thought. Even the rebuttal's author does not argue that the models reason well, only that this experiment measured the wrong thing. And the practical fact sits outside the whole quarrel: the models that "collapse" on twenty-disk Hanoi are the ones people use all day, "already useful today, whether or not they can reliably solve the Tower of Hanoi or River Crossing puzzles."⁶