Google's eye-screening AI rejected a fifth of the photos nurses took in Thai clinics

A screening tool that scored like a specialist

Diabetic retinopathy is damage to the small blood vessels at the back of the eye. Catching it means someone examining a photograph of the retina for the tiny hemorrhages and swelling that mark the disease. There are rarely enough eye doctors to examine every one. Thailand had about 1,500 ophthalmologists and 200 retinal specialists for roughly 4.5 million people with diabetes, one eye doctor for every three thousand patients who might need one.¹ The Ministry of Public Health wanted 60% of diabetic patients screened each year and had not reached half since 2013.¹

A deep-learning model looked like a way to close that gap. In 2016 a team at Google trained a neural network on 128,175 retinal photographs. A panel of 54 ophthalmologists and senior residents graded each image three to seven times, and the majority vote became the answer the model learned to reproduce.² On a held-out set of curated images it reached 90.3% sensitivity and 98.1% specificity for referable disease: it caught most of the eyes that needed a doctor and rarely flagged one that did not.² Measured later against Thailand's own national-program graders, the system cut the false-negative rate by 23%, missing fewer referable cases than the human graders it was compared with.³

On that record, Google Health, working with Rajavithi Hospital in Bangkok and Thailand's Ministry of Public Health, took the system into real screening clinics, where nurses already photographed patients' eyes for a doctor to read later.¹ Between late 2018 and 2020 it was deployed at three clinics in Pathum Thani, and a team of Google researchers visited eleven clinics there and in Chiang Mai to watch what happened when nurses, not benchmarks, put it to work.¹

The images the clinics actually sent

The trouble began at the camera. The system had, in the researchers' words, "stringent guidelines regarding the images it will assess." It "only assesses the highest-quality images," and if a photograph had "a bit of blur or a dark area" it declined to grade the image at all, "even if it could make a strong prediction."¹ This was deliberate, chosen for patient safety: better to return no answer than a wrong one. On the curated photographs the model had trained and been tested on, every image cleared that bar, because lab images are taken to clear it.² Clinic images were not.

Of 1,838 images put through the system in the first six months across the three Pathum Thani clinics, 393 came back ungradable, a little over one in five.¹ Some clinics photographed patients in rooms that could not be fully darkened, so stray light reached the image. One clinic's camera needed repair. Some sites had stopped using the dilating drops that widen the pupil for a clean shot.¹ A report in MIT Technology Review described nurses "scanning dozens of patients an hour and often taking the photos in poor lighting conditions," and put the rejected share at "more than a fifth."⁴

For a nurse, an ungradable verdict meant doing the work again. She would reposition the patient and retake the photograph, two to four minutes each time, and she could not keep trying, because the flash left the pupil too constricted for a clean third attempt.¹ One nurse, faced with an eye the system kept refusing, tried to build a single usable picture from two flawed ones, the top of the retina from one photograph and the bottom from another. The system could not take a composite. It needed one good image per eye, and it had already decided it had none.¹

What the gate cost the room

Ungradable images were the most visible failure. The connection was the next one. The system did its grading in the cloud, so every photograph had to travel there and back, and the clinics' internet was not built for it. On a strong connection a result came back in a few seconds; on a weak one a single image took 60 to 90 seconds to upload.¹ At one clinic a two-hour internet outage cut the patients screened that day from about 200 to about 100.¹

Then there was the referral. A patient the system flagged had to travel to Pathum Thani Hospital, an hour away, for a specialist to confirm the finding.¹ Nurses knew what that trip cost a working patient, and some told patients what taking part could mean before they consented. At one clinic, by a nurse's estimate, 40 to 50% of eligible patients declined to enroll.¹ These were people who had already come to have their eyes checked, opting out of the screening they had come for, because the workflow around the tool made saying yes expensive.

TechCrunch, reviewing the same study, judged that Google's own blog post about it "presents a rather sunny interpretation of events," given documented problems that included the lost throughput and nurses steering patients away from the program.⁵ Google's researchers did change the system in response. Writing on the company's blog, the field study's lead author, Emma Beede, said the team amended the protocol. An eye specialist now reviews an ungradable image and the patient's records "instead of automatically referring patients," to spare them "unnecessary travel, missed work, and anxiety about receiving a possible false positive result."⁶ The full prospective program built the same safeguard in from the start: a retina specialist over-read every image.⁷

Why a good model still failed the clinic

When the full program was run to completion and reported in The Lancet Digital Health, the model's field accuracy for vision-threatening disease was 94.7%, close to the 93.5% of the retina specialists who over-read the same images.⁷ The diagnosis transferred. What did not transfer was everything the diagnosis depended on, and a fifth of the images never reached it.

A benchmark answers one narrow question: given an image the system agrees to score, how often is it right? It says nothing about how many real images the system will agree to score, how long each takes to upload, or how far a flagged patient must drive. The 90.3% sensitivity from the lab was true, and it was also silent about all of that.

The model was trained on curated photographs, taken and selected to be gradable.² The clinics produced photographs shot under fluorescent light, sometimes on a camera that needed repair, by nurses with about ninety seconds per patient.¹ This is distribution shift, the gap between the data a model learned on and the data it meets in the world, which the course has met before in a self-driving crash. Here it wears a second face. The model did not misread the new images. Its quality gate, set high for patient safety, looked at the messier field images and refused a fifth of them. On the images it did grade, the diagnosis held, so the same shift that might have shown up as wrong answers showed up instead as no answer at all.

The comfortable reading is that the clinics simply could not keep up with the technology. The evidence points the other way. An outside analysis by Chinasa Okolo, a researcher unaffiliated with Google, located the failure earlier, in how the tool was brought in. Its verdict: "The lack of understanding around the existing workflows... led to the failures of this AI system once deployed." Studying those workflows, Okolo wrote, "would have benefited the development of the system had they occurred pre-deployment."⁸ The clinics were the environment the system was for. The bench number described the images the model accepted, and nothing about the place that produced them.

Where a clean benchmark hides the gap

That distance sits inside almost any system sold on a single benchmark. A speech-recognition model scored on clean audio meets a noisy room. A risk model checked on one hospital's patients meets another hospital's. In each case the published number is honest about the test and silent about the deployment, and the deployment is where the patients are. The Lancet team, having run the whole program, stated the general rule plainly: "socioenvironmental factors and workflows must be taken into consideration when implementing a deep-learning system within a large-scale screening programme."⁷

What Google's team did that is uncommon was to go and look before the gap surfaced as harm. Most deployments never produce a record of the field at all: the benchmark is published, the product ships, and the distance shows up only as complaints. Two independent experts quoted in MIT Technology Review made that point. Michael Abramoff, an eye doctor and computer scientist at the University of Iowa, said he was glad Google was "willing to look into the actual workflow in clinics." Hamid Tizhoosh of the University of Waterloo called the work "a crucial study for anybody interested in... actually implementing AI solutions in real-world settings."⁴