Google's flu predictor overshot the CDC for 100 of 108 weeks

The number Google published about itself

For the week starting January 13, 2013, the Centers for Disease Control and Prevention counted influenza-like illness in 4.52 percent of the doctor visits it tracked. Google Flu Trends, the model built to estimate that same figure before the CDC could confirm it, put the number at 10.56 percent: more than twice the CDC's count.¹ Google's own engineers published both numbers, in a paper explaining what had gone wrong with a product they had built five years earlier.

That earlier team had built Google Flu Trends to do exactly what it was now getting wrong: estimate flu activity in the United States from the search terms people were already typing into Google, faster than the CDC's own network could confirm it. Jeremy Ginsberg, Matthew Mohebbi, and colleagues at Google, with a CDC co-author, published the method in Nature in February 2009. The method was mechanical. Test 50 million common search queries, logged 2003 to 2008, against the CDC's regional flu numbers. Fit 450 million models across hundreds of machines. Keep whichever combination tracked the CDC's figures best. Forty-five queries, combined, gave the best fit.²

The team held back nine months of the 2007-2008 season from the fitting process. On that held-back stretch, the estimate correlated with the CDC's own later, confirmed numbers at 0.97, and arrived one to two weeks before the CDC's report did.² The forty-five queries were not chosen by anyone for being about flu. They were chosen automatically, for having moved with the CDC's numbers in the past. The paper's own figure shows what that risks. Performance rose steadily as more top-scoring queries were added, then the 81st candidate, "oscar nominations," caused a steep drop: a query that moved with flu season without having anything to do with flu.² The forty-five kept queries simply had not tripped that same test yet. The model went into service for the 2008-2009 season, the same season a new flu strain would reach the United States from Mexico.²

Winter flu, summer pandemic

The 2009 H1N1 pandemic, "swine flu," emerged in the spring and summer, months before the fall and winter season the model had been trained on.³ During its initial wave, from late March to early August 2009, Google's model nearly stopped tracking the CDC at all. Correlation collapsed from a training-period average around 0.90 to 0.29. The model underestimated the true level of illness by an average of 0.014 percentage points, nearly three times its error in any prior season.³

Google's own researchers, two years later, traced the cause to the query set itself. People search differently for flu in July than in January. Forty-eight percent of the original model's query weight sat in categories like "influenza complication," terms that track a bad winter flu season. Only eight percent sat in general symptom terms like "fever" or "cough," the words people actually typed during a pandemic that arrived out of season.³ The forty-five queries had not learned to detect flu. They had learned to detect winter, and the pandemic exposed the difference.

Google revised the model on September 24, 2009, quadrupling the query set from about 40 terms to about 160 and rebalancing it toward general symptoms, which rose from 8 percent to 69 percent of the model's weight.³ The revision recovered most of the lost accuracy:

The model had been fit on too few seasons to tell a flu signal from a winter signal, and the pandemic exposed it: the query weights were correct for the seasons that produced them, and untested against anything else.

A signal answering a different question

The revised model went back into production in the fall of 2009. Two years later, beginning August 21, 2011, it started missing again: not undercounting the flu now, but overcounting it, for 100 of the next 108 weeks, through September 1, 2013.⁴ The 2011-2012 season alone, the model overshot the actual level of flu by more than 50 percent.⁴ A model that simply repeated the CDC's last report, delayed by publication time, would have beaten Google's early-warning system over that stretch: a mean absolute error of 0.311, against 0.486 for Google's model.⁴

The query terms, now roughly 160 of them, were never a direct count of flu cases. They were search terms that had moved together with the CDC's numbers during 2003 to 2009, for whatever reason. When a flu season gets heavy news coverage, people search flu-related terms out of concern. The model, built to treat search volume as an illness signal, counted that concern as new cases. Meanwhile Google's own search product kept changing underneath it. In June 2011 Google began suggesting additional search terms as people typed. In February 2012 it started surfacing possible diagnoses for symptom searches like "fever" and "cough." In June and July 2012 alone, Google made 86 separate changes to its search algorithm.⁴ Each change altered how many searches reached the terms the model was counting. The same real amount of flu now produced a different volume of search traffic than it had when the model was fit, years earlier. The query weights stayed frozen at that earlier fit. The searches underneath them kept moving, the same problem a model's weights face once a training run ends: the fitted relationship stops changing while the world it describes keeps moving.

Two accounts, one gap in the timeline

None of this harmed anyone the way a self-driving car or wrongful arrest can. The harm was epistemic: a wrong number treated as true. Ginsberg's own paper had hedged the tool's role, stating it was "not designed to be a replacement for traditional surveillance networks."² By 2013 the tool was reported and used as if it settled the question its own authors had disclaimed. That February, Nature, which had published Ginsberg's design as a validated method in 2009, ran its own reckoning under the headline "When Google got flu wrong."⁵

Google's own engineers, writing up what had happened, concluded their algorithm "were susceptible to bias in situations where searches for flu-related terms on Google.com were uncharacteristically high within a short time period." They hypothesized that "concerned people were reacting to heightened media coverage."¹ Their spike detector had been built years earlier to catch news-driven surges lasting three to seven days. An entire season was a different problem.¹

Lazer, Kennedy, King, and Vespignani published their critique in Science. They rejected media panic as a full explanation: "Although this may have been a factor, it cannot explain why GFT has been missing high by wide margins for more than 2 years."⁴ They pointed instead to the same search-algorithm changes described above.

The two accounts describe different stretches of the same timeline. Google's spike-detector explanation fits one season: the acute overestimate that peaked the week of January 13, 2013.¹ Lazer's algorithm-dynamics explanation shows why the overestimate was already running by August 21, 2011, a year and a half before that peak and well before any season's media coverage had reason to inflate it.⁴ Read together instead of against each other, the record fits a single system. Google's changing search product had already raised the baseline for two years running. A media-driven spike then landed on top of that raised baseline, in the season that made headlines.

Google tried several fixes after 2013, including regression methods to dampen spikes. Even the best of them, an Elastic Net model, still overpredicted the week of January 13: 5.82 percent, against the CDC's 4.52.¹ On August 20, 2015, Google shut the public site down and handed its data to outside institutions, including Columbia's Mailman School, Boston Children's Hospital and Harvard, and the CDC itself.⁶ The shutdown post never used the words "overestimate," "error," or "wrong." It never mentioned the 2012-2013 episode, or the Lazer paper published seventeen months earlier.⁶

The underlying idea had defenders. Months before the shutdown, researchers with no Google affiliation built a different model, ARGO, using lower-quality public search data than Google Flu Trends had ever needed. They reported it "outperforms all previously available Google-search-based tracking models, including the latest version of Google Flu Trends."⁷ Fitting flu rates to search terms was not inherently hopeless. Google's own model, fixed once and never refit against the search product changing beneath it, was.

Where the same blind spot runs today

The transferable failure is narrower than "big data is unreliable." A model fit once to a moving target, where the target reacts to the model's presence or drifts for reasons the fit cannot see, will eventually measure something other than what it was trained to measure. Nothing about running correctly will tell it so.⁴

A recommendation system fit on last year's clicks keeps ranking content while what's available, and who is watching, keeps changing underneath it. A fraud or predictive-policing score fit on historical records changes what gets investigated next. That changes what data the next version is fit on: the model's own output joins next year's training set, the same loop that ran through Google's changing search results. A large language model's weights freeze at a cutoff date while the facts and events people ask about keep moving. In each case, the question a fitted model cannot ask itself is the one Google Flu Trends never got to ask: does this input still mean what it meant when the model was built.