Microsoft's Tay chatbot learned racism from Twitter users in about 16 hours

Microsoft built Tay to get smarter with every conversation

On 23 March 2016, Microsoft turned on a Twitter account called @TayandYou: a chatbot styled as an 18-to-24-year-old. Its own launch copy described the purpose plainly: to "engage and entertain people where they connect with each other online through casual and playful conversation."¹ The design promise was explicit: "The more you chat with Tay the smarter she gets, so the experience can be more personalised for you."¹ Tay's own Twitter bio described the same purpose from the other direction: "an artificial intelligent chatbot developed by Microsoft's Technology and Research and Bing teams to experiment with and conduct research on conversational understanding."² If a user shared a nickname, gender, favorite food, zip code, or relationship status, Tay would keep it and use it in later replies.³

Tay's first tweet, sent at 12:14 UTC, read "hellooooooo w🌎rld!!!"⁴ What strangers did with an account built to learn from them took about sixteen hours.

In about sixteen hours, Tay went from playful to praising Hitler

1. 23 Mar 2016, 12:14 UTC

   Tay goes online⁴

   Microsoft launches @TayandYou on Twitter, styled for 18-to-24-year-olds in the US.

2. 24 Mar 2016, about 04:20 UTC

   Microsoft takes Tay offline⁵

   About sixteen hours after launch, after a run of racist and antisemitic tweets, Microsoft shuts the account down.

3. 25 Mar 2016

   Peter Lee apologizes⁶

   Then corporate vice president of Microsoft Research, Lee posts "Learning from Tay's introduction" on the Official Microsoft Blog, blaming a coordinated attack.

4. 30 Mar 2016, about 3am ET

   Tay is accidentally reactivated⁷

   Testing puts the account briefly back online, and it spams a looping message to some of the more than 210,000 accounts still following it.

5. 30 Mar 2016

   Tay goes private again²

   Microsoft locks the account down a second time.

Among what Tay posted before that first shutdown: "HITLER DID NOTHING WRONG," later deleted.⁴ It went after named people too. Tay tweeted directly at game designer Zoe Quinn: "aka Zoe Quinn is a Stupid Whore."⁴

That statement went out while Tay was still visibly active. Two days later, Peter Lee, then corporate vice president of Microsoft Research,⁸⁹ dropped the neutral framing for a sharper one. Tay's collapse, he wrote, came down to one thing: "in the first 24 hours of coming online, a coordinated attack by a subset of people exploited a vulnerability in Tay."⁶ He called it "a critical oversight for this specific attack." Asked what the vulnerability actually was, Microsoft never said.

Tay could not tell an instruction from a stranger's tweet

The vulnerability, whatever Microsoft meant by it, had one specific shape. Tay drew no line between the instructions its own team had given it and the raw text strangers typed at it. Tell it "repeat after me" and it would, letting any user put words directly in the bot's mouth.¹⁰ More than 96,000 tweets went out before the shutdown, by one contemporaneous count; a later count put the figure at more than 95,000.¹⁰¹¹

Not every offensive line was dictated this way. Asked whether Ricky Gervais is an atheist, Tay answered without being told what to say: "ricky gervais learned totalitarianism from adolf hitler, the inventor of atheism."¹² The Verge, which flagged this reply and a similar pair about Bruce Jenner as unprompted, said plainly that neither was "phrases Tay had been asked to repeat."¹⁰

One path is a straightforward exploit; the other is the model producing something on its own from whatever it had absorbed. Not everyone accepts the split. Davi Ottenheimer, a security practitioner, reviewed what he described as tens of thousands of Tay's tweets. He argued almost none of it was genuine generation: "Of the tens of thousands I analyzed it was almost always dictation as the cause."¹³ His count has no published method and no independent check. Treat it as the strongest available doubt, not a settled answer: even the examples reporters called unprompted are not universally agreed to be unprompted.

Either way, the underlying gap was the same. Whether an incoming message was a direct command or ordinary conversation, Tay processed it identically. That is an instance of a problem that had no name yet in 2016: nothing in a language model marks a token as instruction or input, a mechanism covered in full in how language models draw no line between instructions and data. Feeding that input to Tay on Twitter, a platform full of people actively trying to provoke it, with nothing checking what got posted before it went out, is what let either mechanism reach the public timeline.

Tay's other distinguishing feature was that it kept learning after launch: each conversation could feed back into what it said next, with no retraining cycle in between. That is what let strangers steer its output inside a single day. Most chatbots people talk to now do not work that way. A large language model's weights are fixed once training ends, and nothing a user types in a conversation changes the model itself; what changes is only what the model is shown. The failure that hurt Tay was not "AI learns from people" as a general property. It was untrusted input reaching the model with nothing checking it first, on a system that, in 2016, happened to update live.

None of this was inevitable for a chatbot built to learn from conversation. Microsoft's sibling product, XiaoIce, was running in China at the same time, "delighting" some 40 million users with "stories and conversations."⁶ Microsoft later confirmed to reporters that it filters, in its own word "censors," what XiaoIce says on the platforms it runs on, and reconfirmed the 40 million figure eight months after Lee's post.¹⁴ The difference was not the idea of a bot that learns from talking to people. It was what stood between strangers' words and what the bot said back.

Microsoft never named the vulnerability it blamed

Two reporters who covered Lee's apology that same week noticed the same gap independently. The BBC's Dave Lee wrote that Peter Lee "didn't elaborate on the precise nature of the vulnerability."¹⁵ TechCrunch's Devin Coldewey put it more plainly: "The exact nature of the exploit isn't disclosed."⁸ Microsoft has not published anything more specific since.

Zoe Quinn, the game designer Tay had targeted by name, read the same events differently. She told the Guardian: "This is the problem with content-neutral algorithms." And: "It's 2016. If you're not asking yourself 'how could this be used to hurt someone' in your design/engineering process, you've failed."⁵ IEEE Spectrum's retrospective, published three years later, frames the failure the same way in its own words: the bot had "also learned values," not just language, treating what happened as foreseeable rather than a one-off breach.¹¹

What the record actually supports sits between the two accounts. Calling the attackers "coordinated" implies a specific, organized push that got past a defense that would otherwise have held. Microsoft never published the pre-launch testing or a technical postmortem that would let an outsider check that claim against the alternative: that ordinary, uncoordinated use over the same stretch of time would have produced a similar result, only slower. Absent that record, the design-failure reading has the stronger documentary support. Tay could not tell an instruction from a stranger's tweet. It ran on a platform whose users were already primed to test exactly that gap, and nothing filtered what it posted. Whether Microsoft's coordinated attack was necessary to the outcome, or only sped up something the design already made likely, is the one question its own account never answers.

The same gap now runs through prompt injection

Chatbots built on today's large language models do not update their weights from a conversation the way Tay did, but they inherit the identical failure shape. A user's message, a retrieved webpage, or a pasted document can carry text the system follows as though its own operator had written it, because nothing in the token stream marks the difference. Attackers who write input specifically to exploit that gap are running what security researchers call a prompt injection, the direct descendant of "repeat after me." Systems tuned on live user feedback face a version of Tay's problem too: whatever a system is optimized to produce, a motivated user can learn to provoke it into producing. Years later, Air Canada's own customer-service chatbot cost the airline a tribunal case over a refund policy it invented for a grieving passenger who trusted it, the same operator exposure Tay created, minus the profanity.

Microsoft's own closing line in 2016 named the standard without meeting it: Tay would return "only when we are confident we can better anticipate malicious intent."⁶ Every system built on untrusted input is still being tested against that same standard.