Epic's sepsis alarm missed two-thirds of the patients it was built to catch

A score wired into the record itself

Sepsis is organ damage caused by the body's own response to an infection spiraling out of control. The international consensus definition, known as Sepsis-3 and agreed by a task force led by Mervyn Singer in 2016, states it plainly: "life-threatening organ dysfunction caused by a dysregulated host response to infection."¹ Hospitals need to catch it within hours, before organs start failing. Epic Systems, the country's largest health-record vendor, built the Epic Sepsis Model, or ESM, for that window. It is embedded directly in the record, scoring every hospitalized patient automatically instead of waiting for a doctor to order a test.² It recalculates a score from a patient's vital signs and labs as they update, and once the score crosses a set threshold, the record posts an alert telling the clinical team to consider sepsis.² Epic marketed it on its own testing: an area under the receiver operating characteristic curve, a number from 0.5 (a coin flip) to 1.0 (perfect), of 0.76 to 0.83.² That is respectable by the standards of hospital prediction tools, and it was generated entirely inside Epic, on Epic's own view of Epic's own customers, before Epic sold the tool nationwide with no published external test.³

The test Epic never ran on someone else's patients

In 2021, a team led by Andrew Wong at the University of Michigan finally ran that test, on hospital data Epic had no hand in choosing. They pulled 38,455 hospitalizations, 27,697 patients, treated at Michigan Medicine between December 2018 and October 2019. Of those, 2,552, or 6.6 percent, involved a patient who developed sepsis.² Scored against what actually happened to those patients, the model's area under the curve came out at 0.63, with 95 percent confidence between 0.62 and 0.64. Not 0.76 to 0.83.² At the model's alert threshold, its sensitivity, the share of true cases it flagged, was 33 percent. It missed 67 percent of the patients who developed sepsis, roughly two out of every three.² Its positive predictive value, the share of its alerts that were correct, was 12 percent. For every eight alerts a clinician chased down, on average one pointed to real sepsis and seven did not.² An editorial in the same issue of JAMA Internal Medicine, by Anand Habib, Anthony Lin, and Richard Grant, put the toll bluntly. A model whose sensitivity and specificity add to less than 1.5, as ESM's 33 and 83 percent do, "must be incorporated into care with caution, particularly when a validation study is not published, as Epic failed to do."³

The gap was not a one-hospital fluke. A 2024 validation at two county emergency departments, a different hospital system and a different set of patients, found the same weakness. Within a six-hour window, sensitivity fell to 14.7 percent and positive predictive value fell to 7.6 percent.⁴ Median lead time was zero minutes: the alert arrived as often after the clinical team had already recognized sepsis as before. Three years after Wong's finding, at a different institution, the model that had scored 0.76 to 0.83 inside Epic was still failing the same way outside it.

The alarm clinicians learned to tune out

At Michigan Medicine, 18 percent of all hospitalizations, 6,971 of 38,455, triggered an alert.² With a positive predictive value of 12 percent, most of those alerts were noise: a clinician who investigated eight of them found, on average, one genuine case of sepsis. That is what alert fatigue looks like from inside a shift. A warning wrong seven times out of eight teaches clinicians to tune it out, so it is not there for the eighth alert, the one that mattered.

The volume made it worse. A separate study across 24 hospitals running the model, led by Andrew Wong, measured alert traffic when COVID-19 arrived. The hospitals' overall patient census fell 35 percent. Sepsis alerts per day rose 43 percent anyway, from 953 to 1,363, and the share of patients triggering an alert on a given day roughly doubled, from 9 percent to 21 percent.⁵ One hospital in the study, the University of Michigan, shut its sepsis alert off within a month of its first COVID-19 admission rather than keep chasing a tool producing more noise as the real workload eased.⁵

The other side of that same 12 percent is the 2,552 people in Wong's cohort who did develop sepsis, two out of three of whom got no warning at all. The alarm flooding clinicians with false positives was, at the same time, silent for most of the true cases.

Epic retrained the model without proving it elsewhere

Reporting by STAT News in 2022 found that Epic had revised the model. Hospitals should now retrain it on their own data before clinical use, rather than run the nationally trained version. The revision also tracks the Sepsis-3 criteria more closely and leans less on treatment orders as an input.⁶ What Epic did not do was rebut the mechanism researchers had identified. Instead, its director of nursing, Emily Barey, pointed to a different, favorable study, one finding faster antibiotic administration and better outcomes "without an increase in harmful clinical interventions."⁷ That answers a different question than Wong asked: whether the alert correctly identifies who has sepsis.

The clearest independent look at version 2 comes from a multicenter validation across four health systems. Its area under the curve ranged from 0.82 to 0.92, well above version 1's 0.63.⁸ Performance varied by hospital, and low positive predictive value with high alert volume persisted at some sites. Retraining locally made the tool a better bet in some hospitals. It did not resolve the deeper problem: a hospital cannot know how the tool will perform on its patients until it tests the tool on its patients.

The clock the model was actually reading

Part of the reason ESM scored so well inside Epic sat inside its own inputs. STAT News reported in 2021 that the model used, among its predictors, whether and when a clinician had already ordered antibiotics or a blood culture, a detail Epic had not disclosed.⁹ An antibiotic order is not a symptom of sepsis. It is a record of a decision a doctor already made, after already suspecting sepsis enough to treat it.

Every prediction model is trained by checking its guesses against a labeled outcome: for ESM, did this patient have sepsis, at a given point in time. The label is only as good as how it draws that line. If part of it depended on treatment a doctor had already ordered, the model could get credit for "predicting" sepsis in cases where the doctor's own order had already given away the answer. Counting the three hours after the clinical team had already recognized sepsis as a hit, the area under the curve rose to 0.80, close to Epic's claimed range.² Score it on the harder question, whether it flagged sepsis before the team recognized and treated it, and the number fell back to 0.63.

That gap sits at the center of a real dispute. The researchers' account: the leakage explains most of the difference between Epic's number and Wong's. Strip out credit for a decision the model didn't make, and performance collapses toward what independent testing found.²⁹ Epic has not published a technical rebuttal of that analysis. Its strongest account is the position it took afterward: hospitals should retrain the model on their own data. That is a claim about patient populations differing across hospitals, not about how the training label was built.⁶ What would settle it, a reanalysis of the model under its training-time definition on an external cohort, is unpublished. Wong's own numbers remain the clearest evidence: scored on the harder definition, the score lost most of the advantage that made it look strong on paper.

The failure ESM exposed is not unique to sepsis, Epic, or medicine. Any score trained against an outcome defined by a professional's own downstream action, a diagnosis entered once treatment has started, a fraud flag set only after an investigator has decided, risks learning to predict that judgment instead of the thing it claims to forecast. A hospital readmission score trained on which patients a discharge planner already flagged for follow-up faces the same trap, and so does a hiring tool trained on which resumes a recruiter already advanced. The question worth asking of any of them is the one Wong's team had to ask of ESM: what counted as the right answer when the model was trained, and whose decision already knew it.