Babylon sold an AI as good as a doctor, and its chatbot read a heart attack as a panic attack

Babylon promised a doctor in an app

Babylon Health was a London digital-health company that put a doctor's first two jobs, telling you what might be wrong and how urgently to act, into a phone. Its NHS service, GP at Hand, let patients in England leave their local practice for Babylon. Its symptom checker, a triage chatbot, asked about symptoms and told the user what might be wrong and where to take it. In June 2018, at an event with the Royal College of Physicians, Babylon presented a study it said showed the system diagnosing and triaging as well as GPs.¹ Reporting on the launch recorded the sharper version: Babylon said its algorithm had beaten doctors on the Royal College of General Practitioners' official exam.²

The study said something narrower than the marketing did. Its own abstract reported diagnostic accuracy "comparable to human doctors" and triage advice that was "on average, safer than that of human doctors ... with only a minimal reduction in appropriateness." Comparable, and safer on average, not better.¹ And in its discussion the paper set a limit on all of it: "our results cannot be directly interpreted with respect to real-world accuracy and safety."¹ Two problems begin here. One is a claim that outran the paper it rested on. The other is a set of real cases that test was never built to catch.

The number came from a hundred scripted vignettes

The comparable-to-doctors result came from a test Babylon's researchers built: a "semi-naturalistic, role-play" exam of 100 clinical vignettes, each modelling one condition in a patient aged 16 or over, written by independent doctors. Seven locum GPs, none of whom had built the system, worked the cases, giving a diagnosis and triage for each, as did the system. A single independent judge scored the differentials and set the acceptable triage range.¹ (Coiera's line-by-line review counts six doctors; the paper's own tables list seven, Doctors A to G.)³

A vignette is not a patient. It is a tidy paragraph that already contains the findings a real clinician would have to draw out of a frightened person. It can only describe a condition the system was built to recognize, because someone wrote it from that list.³ Measured this way the two sides were close. On the main 100 cases the system's diagnostic recall, the share of each vignette's modelled condition it caught, was 80.0 percent, against the doctors' average of 83.9 percent. Their precision and combined scores sat within a point of each other.¹

An average across seven doctors hides how the comparison was built. Enrico Coiera, a clinical-informatics researcher, took the study apart two days after it appeared. Only vignettes inside the system's range were tested, he wrote, so "they only tested Babylon on vignettes it had a chance to get right."³ The doctors had no reciprocal right to exclude the vignettes they were weak on. The averages were also fragile. Removing the weakest of the seven doctors and recomputing, Coiera found, "shows humans beating Babylon on every measure."³ The study also ran no significance test, so it could not say its differences were more than chance.³ Coiera, with Hamish Fraser and David Wong, put the objection into The Lancet that November. The study, they wrote, "does not offer convincing evidence that [it] can perform better than doctors in any realistic situation." There was "a possibility that it might perform significantly worse."⁴

The safety claim carried the same dependence on who was judging. Against the independent judge's range, the system's triage came out 97.0 percent safe to the doctors' 93.1.¹ Scored instead against the strictest of three in-house GPs, the very same triage decisions came out 81.0 percent safe.¹ The yardstick had moved, not the system's answers.

What the vignettes left out

The distance between a prepared case and a real one had already caught medical AI. A Google screening model for diabetic eye disease matched specialists in the lab and then rejected a fifth of the photographs nurses actually took in Thai clinics. Babylon's own gap showed up in what the chatbot told people. In a 2017 test of the checker for WIRED UK, Fraser and Wong found it "performed worst in identifying common illnesses, including asthma and shingles."⁵

The clearest cases came from David Watkins, an NHS consultant oncologist who tested the app in public under the handle @DrMurphy11. Entering classic heart-attack symptoms for a female patient profile, he got back a diagnosis of panic attack or depression; entering the same symptoms as a man returned a possible heart attack.⁶ A fictitious 67-year-old smoker with central chest pain was offered gastritis or a sickle-cell crisis.² These are under-triage: routing a presentation that needs urgent care to something slower. It is the costly error in triage, because the two mistakes are not the same size. Sending a heart attack home is worse than sending a panic attack to A&E.

The regulator had seen the pattern too. In May 2018 the Medicines and Healthcare products Regulatory Agency notified Babylon of two chatbot safety incidents, one missed heart attack and one missed deep-vein thrombosis.⁷ In February 2019 it logged a field safety notice for "Babylon Health: Chatbot Online Doctor."⁸

What none of this gives is a rate. The failures are known through demonstrations named clinicians reproduced and two incidents the regulator logged, none of it with a denominator. Nobody has published how often, in real use, the system under-triaged a serious case. Babylon's own answer was that its AI had been "used millions of times, and not one single patient has reported any harm."⁶ That is not the measurement either, because a self-reported count of complaints is not a safety rate. The record shows that specific serious presentations were reproducibly sent the wrong way, not how often.

Answering the critics by calling them trolls

Babylon defended the system rather than the cases. It described Watkins's work as "Twitter troll tests," called him a "troll," and said his cases were "theoretical data (forming part of an accuracy test and experiment) rather than a genuine health concern from a patient."⁶ Its clinical director, Keith Grimes, said the checker gave "safe information to patients to allow them to seek help in the right timing."² When Watkins raised roughly a hundred cases, Babylon said a panel of clinicians had investigated them and corrected 20 errors in the AI, without saying which.²

The institutions that looked did not read the cases the same way. The Royal College of General Practitioners rejected the exam claim, saying the materials Babylon used to test its algorithm "didn't represent the full range of the test."² In December 2020 the MHRA wrote to Watkins that "Your concerns are all valid and ones that we share."⁷ It added that he had "raised a complex set of issues and there are several aspects that fall outside of our existing remit." The regulator called the concern valid and the rules incomplete.

None of this slowed the company, for a while. Babylon had been valued at nearly $2 billion in 2019.⁹ In 2021 it merged with a blank-check company in a deal that valued it at $4.2 billion,¹⁰ and listed on the New York Stock Exchange that October.¹¹ Two years later it was gone. By August 2023 its US-listed shares were worth a market capitalisation "just over $5,000," and the US business was insolvent. The UK subsidiary went into administration on 31 August 2023, its assets sold to eMed Healthcare UK, which took on around 700,000 UK patients.⁹

Today's symptom checkers still ship without a miss rate

The tools have changed since 2018, but the shape of Babylon's problem now sits behind every chatbot that answers a health question. Consumer symptom checkers and general-purpose assistants give fluent, confident medical answers, and fluency is the easy part. The hard part is the one Babylon's record exposed and its own paper had conceded. A system can look accurate on a curated set of cases and still route the rare, dangerous presentation the wrong way, and that is the failure a demo is least likely to surface. Babylon's own paper said its vignette results could not be read as real-world safety, and that holds for any number drawn from cases a maker picked itself.¹ The gap the MHRA named, that a triage tool could concede a valid safety concern and still sit partly outside the rules, has not closed.⁷