A health algorithm predicted cost, and scored sicker Black patients as lower-risk

The score that chose who got extra care

Health systems use a risk score, a single number predicting how much medical attention a patient will need, to pick who receives extra help. One commercial scoring tool ran at a large academic hospital, rating every primary-care patient in a risk-based insurance contract and flagging the highest scorers for a high-risk care management program that adds nurses, appointments, and follow-up. Patients above the 97th percentile of the score were enrolled automatically.¹

In October 2019 Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan published a study in Science of that tool. They worked from 49,618 patients at the hospital, 6,079 Black and 43,539 white, observed over 100,009 patient-years from 2013 to 2015.¹ The score decided who the program reached.

Hold the score still

Take two patients the tool gave the same risk score, one Black and one white. By the score they are equally sick. They are not. At the 97th-percentile line where patients were enrolled automatically, Black patients carried 4.8 active chronic conditions on average against 3.8 for white patients at the same score, 26.3% more (P < 0.001).¹ The gap holds across the whole range of scores and widens toward the top.¹

In the program as it ran, 19.2% of the patients it enrolled were Black.¹ The authors then asked what that group would look like if the score tracked illness instead. Swapping in the sicker patients the score had passed over, at each threshold, until the marginal patient of each race was equally sick, would raise the Black share from 17.7% to 46.5%.¹ That second figure is a simulation, not a change anyone made. It is the share you reach on paper once the health gap at each score is closed.

The label was cost

A prediction model learns whatever target it is trained on. That target is the label. This tool's label was not illness. It was next-year total medical cost. The tool predicted how much each patient would cost the health system in the coming year, and the program read that cost prediction as a measure of need.¹ Predicting cost is the industry default: the most accurate commercial risk-scoring models predict future spending from past spending, and the Society of Actuaries evaluated the ten most widely used tools by how well they predicted cost.²

Cost and illness are not the same thing, and they come apart by race. At equal illness, less money is spent on Black patients, through worse access, fewer specialist referrals, and less trust built over time, so their recorded cost runs below their need. The wedge is about $1,801 a year at the same number of chronic conditions.¹ The tool learned cost, so it learned that lower spending and scored those patients as lower-risk.

By its own target the tool was well-calibrated. At the same risk score, the two groups' realized costs nearly match.

Cost lined up across race. Illness did not. A tool audited only on how well it predicted cost would pass that audit. And the tool never used race as an input; the paper states plainly that it excludes race.¹ The disparity arrived anyway, carried in by the correlation between spending and race.

The vendor confirmed the bias

The study named no vendor. The authors reported no contact with the tool's manufacturer until the analysis was finished, and the paper identifies the product nowhere.¹ The public code repository they later released keeps the same reticence, calling the source only "our health system partner."³ The name came from outside the study. Reporters identified the tool as Optum's Impact Pro.⁴ On 25 October 2019, the day the study appeared, New York's Department of Financial Services and Department of Health wrote jointly to UnitedHealth Group's chief executive, David Wichmann, naming Impact Pro and calling its racially disparate results "unlawful in New York."⁵ The letter, signed by Superintendent Linda Lacewell and Commissioner Howard Zucker, asked the company to prove the tool was not discriminatory or to stop using it.⁵

The company did not dispute the finding.

That restates the study's own account of the mechanism, offered as a defense. The disagreement is about what the tool should predict, not about whether predicting cost produced the gap. The manufacturer then replicated the analysis on 3,695,943 commercially insured patients of its own and found the same pattern: at a given risk score, Black patients had 48,772 more active chronic conditions than white patients.¹ Relabeling the model to predict health rather than cost alone cut that excess by 84%.¹

When the label is a proxy

The failure here is not special to health care. It is a property of a choice every scoring system makes. When the thing you care about is hard to measure, illness or need or merit, you train the model on a stand-in you can measure. That stand-in is the proxy. The model learns the proxy exactly, and inherits every way the proxy differs from the target you meant.

Cost is one such proxy, and the authors showed the choice of label alone drives the result. Train the same tool on total cost and the highest-risk group is 14.1% Black; train it on the number of active chronic conditions and the group is 26.7% Black.¹ Same patients, same features, nearly twice the representation, from the label and nothing else.

Proxies stand in for goals across deployed scoring: engagement for what a reader values, arrests for crime, prior spending for health need, past hiring for a good candidate. The health tools that predict need this way reach on the order of 200 million people a year, which is what makes the choice of label consequential.¹ In 2021 the study's authors published an Algorithmic Bias Playbook describing biased algorithms as "deployed throughout the health care system, influencing clinical care, operational workflows, and policy," and set out a four-step audit for finding the same fault elsewhere.⁷⁸ The pattern recurs in cases this desk has covered: a credit-limit model New York cleared without proving it fair, another UnitedHealth tool whose length-of-stay estimate set discharge dates, a resume screener that learned to penalize signals of being a woman. In each, the label carried what no one wrote into the features.