Move a tumor detector to a new hospital and it loses twenty-three points

The number belonged to the training hospitals

A tumor detector trained on lymph-node scans from a handful of hospitals learns to find cancer in them well. Send it slides from a hospital it never trained on, the same disease prepared and stained on different equipment, and it quietly gets worse at the one thing it does. Nothing was corrupted and no adversary intervened. The tissue is the same disease, and the labels follow the same rule. Only the hospital changed, and that alone erased nearly twenty-three points of accuracy: 93.2% on held-out slides from its training hospitals, 70.3% on the new ones.¹

Read one way, 93.2% is a property of the model, a measure of how well it finds tumors. Read the way the second number forces, it is a property of the model and the slides together: how well it finds tumors in scans from the specific hospitals that supplied its test set. The first reading is the one a leaderboard invites, and it is the one that fails at the new hospital. Every accuracy is a conditional measurement, accuracy given the distribution it was scored on, and the quantity that decides whether it survives a move is the distance between that distribution and the next one.

Distribution shift is the name for that distance. The i.i.d. assumption, that training and test data are drawn independently from the same distribution, is what lets a held-out score stand in for future performance, and it does quiet work under every benchmark. When the deployment distribution differs from the test distribution, the score is still exact, still reproducible, and no longer about the world the model runs in.

Three ways a distribution can move

A distribution can move in three places, and the difference matters because each breaks a model for a different reason. The inputs can move: the population of things the model sees shifts while the rule linking an input to its label stays put. Photographs from a newer camera, slides from a different scanner, sentences from a later decade. Amos Storkey's treatment of the taxonomy sets the input-moving case apart as covariate shift,² and Hidenori Shimodaira named the condition in 2000: the input distribution P(x) differs between the training sample and the target, while the conditional P(y|x) that maps an input to its label is held fixed.³

The base rates can move: the mix of labels changes while each label still produces the same kinds of inputs. A disease grows more prevalent, so a clinic sees more positive cases, though a positive case still looks the way it always did under the microscope. This is prior probability shift, where P(y) changes while P(x|y) is held fixed.² The plainer name for it is label shift; the two words point at the same object.

The rule itself can move: the same input now warrants a different label. A transaction pattern that meant fraud last year is ordinary this year, the inputs identical and their meaning changed. This is concept shift, where P(y|x) changes while P(x) is held fixed,⁴ and it is the hardest of the three, because the thing the model was trying to learn is the thing that moved. It is a static difference between two fixed distributions, distinct from concept drift, the version that unfolds over time in a running stream.

The tumor detector's failure sits in the first category. The disease and its appearance did not change; the hospitals, their scanners, and their staining did, moving P(x) while the labeling rule held. Keeping the three apart is the first use of the concept, because a correction for one does nothing for another, and a benchmark that mixes them reports a single number for three separate problems.

Shift you can build and watch

Once the motion is named, it can be built on purpose and measured. Dan Hendrycks and Thomas Dietterich took the ImageNet validation images and applied fifteen common corruptions, grouped as noise, blur, weather, and digital artifacts, at five severity levels each, seventy-five conditions in all.⁵ Every corruption is a deliberate move in P(x): the labels are untouched and the pixels degraded the way a real camera or a foggy day would degrade them. Accuracy falls across the board, and it falls further for some corruptions than others, which is why they scored models by an error averaged over corruption types and normalized so that an AlexNet baseline reads 1.00.⁵

The move need not be synthetic. Benjamin Recht and colleagues rebuilt ImageNet's test set from scratch, following the original collection recipe, and a broad range of models lost eleven to fourteen points on the fresh set with no corruption at all.⁶ The images were new, so nothing had leaked from years of tuning against the old set. The drop was distribution shift, not a test set worn smooth by reuse, and the signature that rules the second explanation out is in the numbers: gains on the old test set carried to the new one along a line steeper than one-to-one, so a model better on the original was reliably better on the replacement.⁶

The same gap appears in deployment data that no one designed to be hard. The WILDS benchmark collects ten real-world tasks where training and test distributions come apart the way they do in practice, across different hospitals, cameras, countries, and years, and trains standard models on each.¹ The out-of-distribution score sits below the in-distribution score everywhere.

Species recognition on iWildCam falls from 47.0 to 31.0 in macro F1 when the camera traps change location, and land-use classification on FMoW falls from 48.6% to 32.3% in its worst region.¹ That gap is what the concept buys: a place to look for failure before deployment finds it, and a procedure for measuring it. Hold out a slice of the world the model has not seen, and score there.

The gap you cannot compute, and the ranking you can

The procedure carries a cost the tumor detector already showed. To measure the gap you need the target distribution, a labeled sample from the deployment world. Without slides from the new hospital the 70.3% is unknowable, the 93.2% is all you have, and the 93.2% is the number that misleads. That is the sense in which a benchmark number stops being knowable: the moment the deployment distribution departs from the test set, the size of the departure is exactly what the test set cannot report.

A qualification cuts against the idea that shift is a free predictor of failure. John Miller and colleagues measured out-of-distribution against in-distribution accuracy across many models and many shifts and found them tightly, linearly related: a model's rank under shift is often predictable from its ordinary accuracy alone, more precisely than domain-adaptation theory expected.⁷ The ordering of models is frequently free. You do not need the target distribution to guess which model will lead under shift. What is not free is the magnitude, how far every model falls together, and only the target distribution sets that.

The distinction reframes robustness. When a training trick is reported to improve robustness, the question is whether it moved a model above the in-distribution trendline or only along it. Rohan Taori and colleagues evaluated 204 ImageNet models across 213 shifted test conditions and found that most interventions buy no robustness beyond what their in-distribution accuracy already predicts; the main thing that helped was training on larger and more diverse data, and even that did not close the gap.⁸ Purpose-built methods for shift, tested across the WILDS tasks, leave a large gap between in- and out-of-distribution accuracy standing too.¹ A robustness number that only tracks the trendline is reporting better in-distribution accuracy under another name.

The line that makes ordering free also breaks, and where it breaks it breaks hard. Miller's own exceptions include some CIFAR-10-C corruptions and, again, Camelyon17: cases where in-distribution accuracy stops predicting the out-of-distribution score at all.⁷ The tumor detector is both examples at once. It is where the concept pays, since the twenty-three-point gap named a real failure before a patient met it, and it is where the cheap prediction fails, since on this data ordinary accuracy would not have warned that the fall was coming. Knowing which of the two regimes a system is in is the concept's real return, and that knowledge costs a labeled sample of the target like everything else here.

Two claims inside one accuracy

A benchmark score names one model measured against one distribution, and it carries two claims that come apart under a move. The ordering it implies, that this model beats that one, often travels, and ordinary accuracy can vouch for it without the target distribution in hand. The margin it implies, that the model is this good, travels only as far as the test distribution reaches. The moment the deployment world differs, the margin is not a fact about the model but a fact about the pairing of model and test set that produced it.

Which of the two a given number is offering is worth settling before trusting it, and settling it means asking what distribution the model was scored on and how far that sits from the one it will meet. The tumor detector answered 93.2% to the first distribution and 70.3% to the second.¹ Both are exact. Only one of them is about the hospital where the next scan is read.