Mamba dropped attention to make long sequences cheap. Its authors put some back

The paper people mean by "post-transformer"

A transformer answers each new token by comparing it against every token that came before it. Mamba, a sequence model published in December 2023 by Albert Gu of Carnegie Mellon and Tri Dao of Princeton, answers without looking back at all. It keeps a single fixed-size summary of everything read so far and updates that summary one token at a time, so the work it does per token never grows with how much it has already read.¹

The pitch is cost. Because a transformer holds the entire past in reach, its bill grows with the length of the sequence; Mamba's does not. The paper reports four to five times the inference throughput of a transformer its size, and a scaling curve that keeps pace with a strong transformer recipe over the range it tested.¹ Two years on, those numbers are what people reach for when they argue attention is finished.

The state that stands in for the past

A state-space model keeps a state: a fixed-size list of numbers that stands in for everything the model has read. At each new token it performs a recurrence, which just means it produces the next state from the previous state and the token in front of it, then reads its answer off that state. In the simplest linear form the update is a weighted mix,

where x_t is the current token, h_t the state after it, and A, B, C are the fixed sets of weights the model learned.¹ Read the two lines plainly: carry a little of the old state, fold in a little of the new token, and let the state be the model's entire memory of the sequence.

Set that beside attention. Attention answers a token by reading every earlier token again. A state-space model answers it by reading one thing: the state. The paper puts the contrast in terms of compression. Attention does not compress the past at all; it keeps every earlier token around and re-reads them, which is why its cost climbs with length. A state-space model compresses hard: everything it will ever know about the past has to fit inside that one fixed state.¹ That buys the constant per-token cost. It also makes the state a bottleneck: whatever will not fit in it is lost, and what will not fit turns out to decide how far the model can go.

What "selective" changed

State-space models were not new in 2023. The line Mamba builds on, the S4 family from 2021, already carried a state forward and was strong at very long inputs, solving a 16,000-step benchmark that earlier models failed.² But S4's update was fixed: the same weights applied to every token, whatever the token was. That constraint had a payoff. When the update never changes, the whole recurrence can be unrolled and computed in one parallel pass, like a convolution, which trains fast. Its price was that a model treating every token identically cannot decide that some tokens matter and others are noise.

Mamba's one real change is to let the update depend on the token in front of it. This is what the paper calls selection: how much of the new token to write into the state, and how much of the old state to keep, is now computed from the token itself. Shrink the state to a single number and the update becomes a plain gate.

That gate is the entire idea at its smallest. The model looks at each token and chooses whether to remember it. Making the update depend on the token breaks the convolution trick S4 relied on, so Mamba has to run as a true step-by-step recurrence. Much of the paper's engineering goes into keeping that recurrence fast, by holding the state in the GPU's quickest memory rather than writing it out and back.¹ The payoff shows on a task built to need selection: copy only the tokens that were marked and skip the rest. The fixed S4 scored 18.3 percent; selective Mamba scored 99.8.¹ Choosing what to keep is what closed that gap.

The 2.8B behind the headline

The line everyone quotes is that Mamba "matches Transformers twice its size." The largest language model the paper actually trained was 2.8 billion parameters; the "3B" of the abstract is that same model rounded up. The "twice its size" claim rests on one comparison: zero-shot common-sense reasoning. Across six tasks, Mamba-2.8B averaged 63.3 against 59.1 for a same-size Pythia transformer. It also came in ahead of the 61.7 scored by a Pythia nearly two and a half times larger.¹ On perplexity, a standard score for how well a model predicts the next token where lower is better, Mamba's curve tracked a strong transformer recipe over the sizes tested. Those are real results, and they are short-context results at small-to-mid scale.

The authors are the ones who say so.

The categories most in question at this point, long-context retrieval and in-context learning at frontier scale, were not run at language-model scale. The throughput headline needs the same care. The four-to-five-times figure is generation throughput on one GPU, and it comes from a specific advantage: with no stored past to hold, Mamba runs far larger batches than a transformer of the same size can. In training, its scan only overtakes a fast attention implementation once the sequence passes roughly 2,000 tokens.¹ The win is real and it is specific: long-sequence, large-batch generation, not a blanket speedup.

Where the successor story breaks

Because everything a fixed state knows must fit inside it, a state-space model struggles with any task that needs the exact earlier text: copying a long string, or looking a specific entry up in a long context. A group at Harvard proved the limit and then measured it. Transformers learn to copy with about a hundred times fewer training examples than the best state-space model. On a phone-book lookup, even a 410-million parameter transformer beats a 2.8-billion parameter Mamba once the book is long enough.³ The detail that matters most: this held even though those Mamba models had slightly lower perplexity. "Matches on perplexity" and "matches on the thing you would use it for" are different claims, and Mamba's headline was the first kind.

A year on, Gu and Dao's follow-up sped up the layer and then put attention back in. A model with about a tenth of its layers being attention beat both a pure state-space model and a strong transformer. They credit the gain to attention's knack for retrieving an earlier token instead of forcing everything through the compressed state.⁴ Industry landed in the same place. Jamba, billed as the first production-grade hybrid, interleaves attention and Mamba layers, and its makers state plainly that state-space models like Mamba "still lag behind the performance of comparably sized Transformer language models."⁵

Falcon Mamba, a 7-billion parameter model with no attention at all, reports beating Mistral 7B and Llama 3.1 8B on standard leaderboards.⁶ A pure state-space model really can be competitive at that size, and this is the honest high-water mark for one. But 7B is well short of frontier scale, and those leaderboards are the short-context reasoning tasks where state-space models were always competitive. The result does not test the copying and retrieval gap, and it does not close it.

Mamba showed that a fixed-state model can match a same-size transformer on short-context reasoning up to 2.8B, and generate long sequences more cheaply. It did not show that a pure state-space model matches transformers on retrieval, at long context, or at frontier scale, and its own authors keep some attention for the tasks where the fixed state falls short. Two years after the paper, a technical explainer from IBM records the plain state of things. Transformers "have remained the dominant mode of LLM," and "still outpace both Mamba and Mamba-2 on tasks requiring in-context learning ..., copying, or long-context reasoning." It points to hybrids, its own production models among them, as the way to pair Mamba's efficiency with attention's recall.⁷