Flamingo beat fine-tuned models on six benchmarks with 32 examples

A few examples did the job a labeled dataset used to do

In 2022, DeepMind published Flamingo, a model that takes images, video, and text as input and produces text, and that can pick up a new visual task from a handful of examples placed directly in its own prompt. On six of sixteen published benchmarks, that few-shot version beat systems fine-tuned specifically for the task, using 32 examples and no update to any of its weights.¹

That trick already had a name in text. Two years earlier, GPT-3 had shown that a language model could learn a new task from a few examples typed into its prompt, with no retraining at all, a capability the field calls in-context learning. Flamingo does the same thing, except the examples can include pictures and short video clips as well as sentences. The task it learns can be answering a question about a photograph, captioning a video, or picking the right one of several statements about an image.

Two prior lessons on this desk mark what Flamingo is not. CLIP scores whether an image and a caption already match, a fixed judgment with no room to teach it something new. Segment Anything returns a mask with no name attached to it. Flamingo does neither. Shown a few image-and-caption pairs, it writes new text: a caption, an answer, a description, whatever those examples establish the job to be.

It manages this by joining two pretrained models that never change during Flamingo's own training. One is a vision encoder that turns an image or video frame into a set of vectors. The other is a large language model that turns vectors and words into more words. Two new parts, trained specifically for this job, sit between them: a Perceiver Resampler and a set of gated cross-attention layers, so the language half can read what the vision half saw.¹ DeepMind's own announcement framed the news not as a benchmark record for its own sake but as an efficiency result: "in several cases, the same Flamingo model outperforms methods that are fine-tuned and optimised for each task independently and use multiple orders of magnitude more task-specific data."²

It's easy to read that framing as Flamingo inventing the trick. It didn't. A year earlier, a DeepMind paper called Frozen had already shown that freezing a language model's weights and training only a small bridge to feed it images produced multimodal in-context learning. It did this at a much smaller scale: a single frozen 7-billion-parameter model, one image at a time, no video.³ Flamingo's news was not the idea. It was making the idea work at the scale of a large language model, across video as well as still images, on sixteen benchmarks at once.

Two frozen models and 10 billion parameters of bridge

Flamingo's own architecture breaks into three moves, always in the same order:

1. The vision encoder looks first

   A frozen network called NFNet-F6 turns each image or video frame into a set of vectors. It was already trained before Flamingo existed, and Flamingo's own training never changes it.¹

2. The Perceiver Resampler compresses what it saw

   A new, trained component takes the vision encoder's output and turns it into a small, fixed set of visual features, regardless of how many images or video frames came in.¹

3. Gated cross-attention lets the frozen language model check it

   New layers, inserted between the language model's own frozen layers, let it attend to those visual features while it writes. In the largest model, one of these layers sits before every seventh language-model block.¹

Those two new parts are trained long before any prompt reaches them. Flamingo's makers built them on a purpose-scraped haul of web data: a 43-million-document corpus that keeps images and text in the order a web page actually mixes them, plus roughly 1.8 billion separate image-and-caption pairs bought in from elsewhere.¹ The 32 examples a reader keeps hearing about arrive only afterward, at evaluation, standing in for a task-specific training set the model itself never needed.

The language model being bolted onto is Chinchilla, a 70-billion-parameter model DeepMind had trained separately on 1.4 trillion tokens. At that training budget, it already beat larger contemporaries like Gopher (280 billion parameters) and the original GPT-3 (175 billion).⁴ The vision encoder plays the same role a Vision Transformer plays elsewhere in computer vision, turning a picture into vectors a later model can use, though Flamingo's own encoder, NFNet-F6, is a convolutional network rather than a transformer.¹

Add up the largest model's parts and the split is deliberate. Of the 80-billion-parameter total, 70 billion sits in the frozen language model and 435 million in the frozen vision encoder. The rest, about 10.2 billion parameters, roughly 13 percent, is the Perceiver Resampler and the gated cross-attention layers combined. That 80-billion-parameter version is the one the paper calls simply "Flamingo." Two smaller versions, at 9.3 billion and 3.2 billion parameters, share the same design at a smaller scale.¹

Six wins, nine losses, one benchmark with no comparison

Flamingo's own tables give the shape of that headline result plainly:

Sixteen benchmarks covered captioning, visual question answering, visual dialogue, and multiple-choice tasks built on images and video.¹ One of them, a compositional action-recognition task called RareAct, has no fine-tuned comparison to beat, so it sits outside the count. Of the other fifteen, Flamingo's 32-shot version beat the prior fine-tuned state of the art on six and trailed it on nine.¹

The number in parentheses is how many labeled, task-specific examples the fine-tuned comparison needed. On OKVQA, that is 10,000 against Flamingo's 32, about 310 times fewer. The paper's own summary rounds this to "around 1000 times less task-specific training data," but the real ratio moves with the task: from about 190-to-1 on iVQA up to roughly 15,600-to-1 on the two tasks needing half a million examples, COCO and VATEX. Both are benchmarks where Flamingo's few-shot version did not beat the fine-tuned system.¹

Table 1's own caption, on the same page as this count, prints "seven" instead of six. The paper's introduction and the sentence in Section 3.1 both say six, and the caption on its own overview figure agrees. Recomputing the win-loss column directly from Table 1's own scores gives exactly six. This piece uses six.¹

Two of those six wins are not quite a cold open. OKVQA and MSVDQA belong to a five-benchmark "DEV" set, along with COCO, VQAv2, and VATEX. The authors used that set while choosing the model's own design and hyperparameters, and the paper itself warns that "performance estimates on the DEV benchmarks may be biased, as a result of model selection."¹ That does not undo the win. It does mean two of the six were not run cold.

Even with its weights unfrozen and fine-tuned, on an unlimited annotation budget, Flamingo did not close every gap. Fine-tuning set a new state of the art on five more of the nine tasks it had lost few-shot: VQAv2, VATEX, VizWiz, MSRVTTQA, and HatefulMemes. On the other four (COCO, YouCook2, VisDial, and TextVQA), a fine-tuned Flamingo still trailed the prior fine-tuned state of the art. The paper attributes part of that gap to task-specific tricks it did not use, like optimizing directly for the CIDEr captioning score or fine-tuning on dense annotations for visual dialogue.¹

The frozen-backbone design did not become the field's choice

Flamingo's own weights were never released. Two outside teams confirm the fact rather than each other. The team that later built an open reproduction states plainly that "these autoregressive vision-language models are closed-source, and their weights, training data, code, and hyperparameters are proprietary." A survey of the field independently calls Flamingo "the closed-source vision-language model... by DeepMind" when describing a separate reproduction of it.⁵ Neither the paper nor DeepMind's own announcement says why. The paper's own discussion of risks, in the section built for exactly that conversation, weighs what a model like this could be used for; it does not weigh the decision to keep this one closed.¹ That reason, if one exists on the record, is not in either document.

The closest anyone outside DeepMind has come is an open reproduction called OpenFlamingo, trained on public data (LAION-2B and a public interleaved-webpage dataset) in place of Flamingo's own proprietary sources. Averaged across seven evaluation datasets, its 3-billion and 9-billion-parameter versions reach 85 and 89 percent of their same-size Flamingo counterparts.⁵ The gap is not even. At zero shot, OpenFlamingo-9B trails Flamingo-9B by 6.9 percentage points on OK-VQA and 7.8 points on TextVQA. That gap widens rather than closes as more examples are added to the prompt.⁵ Nobody outside DeepMind has run the paper's own numbers directly. The public version gets close, unevenly.

The design that produced those numbers did not become the field's lasting choice either. A 2024–2025 survey of vision-language models credits Flamingo with the Perceiver Resampler and gated cross-attention design. It also treats Flamingo as the closed reference point later open efforts were built to reproduce, but it documents what came after: BLIP-2 later beat Flamingo-80B on zero-shot VQAv2 by 8.7 percentage points while training far fewer parameters. A more recent model called Sphinx deliberately unfreezes the language model during pretraining, on the reasoning that it improves vision-language alignment beyond what a frozen one allows.⁶ The two frozen halves that made Flamingo's few-shot result possible were, within a couple of years, the part of the recipe later models chose to change.