Every context window has a second number, and it is smaller

The number on the box

When Google introduced Gemini 1.5, it reported that the model could recall a planted fact with better than 99 percent accuracy across as much as 10 million tokens of text, far past the 128,000 tokens of GPT-4 Turbo or the 200,000 of Claude 3.¹ A token is the unit a model reads, roughly three-quarters of an English word, so 10 million tokens is something like 7.5 million words, a small library. This is the context window: the largest amount of text a model will accept in one go. Labs now print it on the box the way a phone prints its storage, and the numbers have climbed from a few thousand tokens to one and two million in three years.

The window is a real limit and a real achievement. It is also two different numbers wearing one label. The first is how much text the model will take. The second is how much of that text it uses well. The second number is measured by separate tests, and it is smaller.

How the ceiling gets set

The window is not a setting someone types in. It is fixed by two things: how the model marks the position of each word, and how long the texts were that it trained on.

Take position first. A model reads all of its input at once, so it needs a way to know that the fifth word comes after the fourth. Most current models use rotary position embedding, or RoPE, which marks a token's position by rotating its representation by an angle that grows with the position.² Word one gets a small rotation, word one thousand a much larger one, and the math lets the model feel how far apart any two words sit.

Those rotations are only practiced up to the longest texts the model trained on. Feed it a longer input and the angles run past anything it saw during training, where its accuracy falls apart. So the advertised window is, at bottom, the longest sequence the position scheme and the training data were built to handle.¹ It is a ceiling the lab sets, then reports.

Moving the ceiling after the fact

That ceiling can move. In 2023, researchers at Meta showed a method called position interpolation: instead of letting positions run off the end of the trained range, squeeze them back into it. A word at position 8,000 is relabeled as position 500, inside the range the model already knows.³ With this, they stretched LLaMA's window from 2,048 tokens to 32,768, a sixteenfold jump, using fewer than 1,000 steps of extra training.³

This changes how to read the number. A context window is not a fixed property of a model that its makers discover. It is partly an engineering choice: train a little more, rescale the positions, and announce a larger number. The window is set and then extended, which is why the same base model can ship with one length this month and a bigger one the next.

The easy test everyone passes

So how does anyone check whether a model uses a two-million-token window? The most common test is called needle in a haystack, built by Greg Kamradt. Plant a single out-of-place sentence, the needle, somewhere inside a long block of filler text, the haystack, then ask the model to repeat it back.⁴ Slide the needle through every depth and every length, and you get a grid of how often the model finds it.

Models pass this test almost perfectly. Gemini 1.5's better-than-99-percent recall across 10 million tokens is a needle-in-a-haystack score.¹ But finding one planted sentence is the easiest thing a long context can ask for. The needle is written to stick out, and the model only has to copy it, not use it. The team behind a harder benchmark, RULER, said so plainly: the needle test measures “only a superficial form of long-context understanding.”⁵

Retrieving a fact is one narrow slice of what people do with a long context. They also summarize across a whole document, answer questions that need two passages combined, and follow instructions buried in the middle. LongBench, a benchmark that scores exactly these jobs, spreads across twenty-one datasets and six task types for that reason.⁶ A single retrieval number says little about any of them.

The number measured separately

So the field measures the second number directly. RULER runs 17 models through 13 tasks at growing lengths and finds where each one stops performing well. The result is stark: every model tested claims a window of at least 32,000 tokens, but only half still perform acceptably once the input actually reaches 32,000 tokens.⁵ That cutoff is RULER's own bar rather than a hard wall, but the pattern holds across the field: the length a model can use trails the length it will accept.

Where inside the window a fact sits matters too. In “Lost in the Middle,” researchers moved the same needed fact to different positions in a long input and watched accuracy trace a U: highest when the fact sat at the very start or the very end, lowest when it sat in the middle, even for models built for long context.⁷ A summary that quietly drops the middle of a document is not a worry someone invented. It is the direct cost of this effect, paid by anyone who trusts the summary without checking it.

Those studies used the models of 2023, and it is fair to ask whether newer ones fixed the problem. They did not. In 2025, the research group at Chroma ran 18 current models, among them GPT-4.1, Claude 4, and Gemini 2.5, and found the same thing: “their performance grows increasingly unreliable as input length grows,” even on simple tasks.⁸

None of this makes the advertised number a lie. Gemini really does retrieve a planted fact at 10 million tokens. The point is narrower and more useful: a model that accepts a length is not promising to reason across it, and the gap does not show up on the spec sheet. It is why people who work with these models day to day treat a big context claim as something to test rather than take on faith.⁹