Segment Anything returns a mask and never a name

What SAM hands back is a shape with no name

Click once inside a dog in a photo, or drag a rough box around a street sign, and the Segment Anything Model returns the precise set of pixels that make up that object. That set of pixels is a mask: the object cut cleanly out of its background, edges and all.¹ What the model does not return is the word "dog" or the word "sign." A mask from this model carries no class label of any kind. The people who built its training data deliberately attached none.¹

The click and the box are prompts. The paper's central idea is a task it calls promptable segmentation: return one valid mask for any prompt, whether that prompt is a point, a box, or a rough mask sketched by hand.¹ A prompt here is a place, not a request in words. Point at a spot and the model outlines whatever is there. It does not read a description and go find the object, and it does not tell you what the object was. Naming, the paper is clear, is somebody else's job.

One heavy encoder, one decoder light enough for a browser

The model is built in three parts.¹ An image encoder does the expensive work. It is a vision transformer, pre-trained by having it reconstruct images with patches blanked out, and it reads the whole photo once into a dense numerical summary. A small prompt encoder turns your point or box into a handful of numbers. A lightweight mask decoder takes the image summary and the prompt together and draws the mask.

Splitting the work this way is what makes the model feel live. The photo is summarized only once. After that, each new prompt runs through the small decoder in about 50 milliseconds inside a web browser, fast enough that dragging a point around updates the outline as you move.¹ A single prompt is often ambiguous. A point on a shirt could mean the shirt, the person wearing it, or both. Rather than commit to one guess, the model returns several masks at once and scores each, and the tool around it picks or lets you choose.¹

The people drop out over three stages

The released dataset, called SA-1B, holds roughly 1.1 billion masks across 11 million images, about 100 masks on every image.² No team of annotators drew a billion masks. The model drew almost all of them, in a three-stage data engine where a person did less work at each stage than the stage before.¹

In the first stage, annotators clicked and corrected while the model was retrained six times on their work. The time to mark one mask fell from 34 seconds to 14, and the masks on an average image rose from 20 to 44.¹ In the second stage the model proposed masks on its own and people only drew the ones it had missed, pushing the average to 72 masks an image and the running total to 10.2 million.¹ The third stage removed the person. The team pointed a regular grid of points at all 11 million images and kept the masks the model returned, which is how the count jumped to 1.1 billion.¹ Of the released masks, 99.1% come from that automatic stage. The 10.2 million a person ever touched are the remaining 0.9%.¹ Meta puts the result at 400 times more masks than any earlier segmentation dataset, built 6.5 times faster than drawing COCO's masks by hand.³

The accuracy of those billion automatic masks rests on a single check: a study Meta ran on 500 sampled images, comparing the automatic masks to careful corrections by experts. On that sample, 94% of the mask pairs agreed on more than 90% of their area, measured by intersection over union, the share of two masks' combined area where they overlap.¹ That is a real number, and it covers 500 images out of 11 million. No independent audit of the full billion masks exists in the record.

Handed the boxes, SAM still trails the specialist

The paper's headline promise is zero-shot transfer: point the trained model at a task it was never trained for, add no new training, and see how it does.¹ It runs five such tests. The most telling is instance segmentation, the job of outlining every separate object in a scene. The model cannot find the objects itself, so the authors hand it the boxes from a separate object detector, ViTDet, and ask it only to fill in the mask inside each box.¹ On the field's standard score, average precision, it loses to that same detector's own specialist model on both benchmarks.

The paper argues the gap is partly the metric's fault: COCO's own reference masks are rough, the specialist has learned to match their roughness, and human raters shown both actually preferred the model's masks.¹ The number favors the specialist; the paper contests what the number measures. But the setup had already conceded the real point: the model never picked a single object. The detector supplied every box and every class, and the model only colored inside the lines it was given. The same gap between prose and table shows up in the other tests. On generating object proposals it trails on the all-objects average, 59.3 against 63.0, even while it leads on the rare and medium ones.¹ On edge detection it lands near the decade-old HED and below the current EDETR, which the authors report as an emergent ability well short of the record.¹

Semantic and panoptic segmentation are the tasks that do put a class name on every pixel: this region is road, that one is car. Those are exactly the tasks the authors say they do not know how to make the model do. The boundary is not a critic's reading of the paper. It is the paper's own account of where its model stops.

The medical wins came after fine-tuning, not before

Meta launched the model as a step toward "the first foundation model for image segmentation."³ Independent coverage ran with it, calling the release a foundation model for vision and "a GPT-esque moment," and slotting it into labeling pipelines as a fast first-pass tool. That same coverage noted the free-text prompting the paper showed off was never actually shipped.⁴ From there the framing hardened into a claim the paper never made, that segmentation was now solved.

Medical imaging is where that claim got tested at scale, because it is full of images unlike the everyday photos the model was trained on. One independent team assembled a benchmark of 1,050,311 images with about 6 million masks across 18 imaging types, ran the original model on it with no fine-tuning, and got a mean Dice score of 58.52%.⁵ Dice is another overlap score, close cousin to the one from the quality study, and 58.52% means the automatic masks and the true ones agreed on a little over half their area on average. On many of those datasets the model simply failed, and specialist models beat it.

The gap closed only when someone retrained the model on the new domain. MedSAM, a version fine-tuned on 1,570,263 medical image-mask pairs spanning ten imaging types and more than 30 cancer types, reports better accuracy than the modality-specific specialists it was measured against.⁶ Zero-shot, the model trailed. Adapted, it led. That is the honest shape of a foundation model: a strong place to start that you retrain for your problem, not a system that already knows it.