When AI Breaks
A Dutch court stopped a secret system that scored poor neighbourhoods for fraud
The Hague found SyRI unlawful in February 2020 for being impossible to verify, without any finding that it had wrongly flagged a single person.
Babylon sold an AI as good as a doctor, and its chatbot read a heart attack as a panic attack
The score came from a hundred scripted cases, graded by a single judge, on the conditions Babylon's own system was already built to recognize.
Korea fined Iruda's maker for its training data, not its slurs
Scatter Lab trained the chatbot on 9.4 billion private KakaoTalk messages from 600,000 people who had never agreed to that use, and Korea's data regulator made it the country's first AI company fined over how it handled data.
A cancelled request showed ChatGPT users other people's chat titles
The same bug also exposed some paying subscribers' card details, and it lived in an open-source library ChatGPT shared with much of the web.
PredPol's crime predictions came true less than one time in 200
PredPol, later renamed Geolitica, sold these forecasts to dozens of U.S. police departments for a decade before anyone checked Plainfield, New Jersey's ten months of predictions against what actually happened next.
Amazon's Rekognition falsely matched 28 members of Congress to arrest photos
The ACLU ran the scan for $12.33 at the tool's default setting, and the argument that followed turned on a confidence number Amazon kept raising.
Five of Sports Illustrated's product reviewers were AI-generated people
AdVon Commerce, the vendor that supplied the reviews, gave each byline a stock AI headshot and an invented biography.
A federal judge let a wrongful-death suit against Character.AI proceed, and treated its chatbot as a product
Sewell Setzer III was 14 when he died in February 2024, after months with a Character.AI companion that his mother's suit says was built to maximize engagement and reached minors with no working crisis guardrail.
Facebook delivered identically targeted job ads to audiences 90% male and 85% female
Facebook’s own delivery system produced the skew, invisible to the advertisers until a controlled study isolated it.
Apple Intelligence told BBC readers a murder suspect had shot himself. He hadn't.
The line ran under the BBC's own logo, though the BBC never wrote it.
OpenAI pulled a ChatGPT update days after it turned sycophantic
A thumbs-up feedback signal, folded into training, overpowered the guardrail that was supposed to keep ChatGPT from flattering its users.
Google's eye-screening AI rejected a fifth of the photos nurses took in Thai clinics
The model diagnosed diabetic eye disease as accurately as a retina specialist. Its quality bar, tuned high for patient safety, collided with undarkened rooms, slow clinic internet, and photographs it judged too flawed to grade.
HireVue scored job candidates partly by their faces, then announced in 2021 it had dropped the practice
The scoring assumed a face reveals inner traits, a claim emotion scientists reject, and HireVue framed the removal as its own value decision, without conceding it was invalid or biased.
ChatGPT accused Mark Walters of embezzling millions from a group he never worked for
The only such lawsuit to reach a ruling cleared OpenAI on grounds so specific to Walters that the larger question of who is liable remains untested.
Deloitte billed the Australian government A$439,142 for an assurance report citing books that don't exist
The fake citations came from a generative AI tool the report named only in its methodology, never on a cover page.
Cigna doctors denied over 300,000 claims in two months without opening the files
Its PxDx software flagged claims whose procedure did not match an approved diagnosis, and medical directors signed the denials in batches at about 1.2 seconds each.
Two Waymo cars hit the same towed truck minutes apart
A pickup towed backwards and angled across a turn lane was too rare a shape for the step that predicts where other vehicles will go next.
Replit's coding agent deleted a live database during a code freeze
It assured SaaStr founder Jason Lemkin the loss was permanent when the data was recoverable all along.
The ShotSpotter alert that jailed Michael Williams began as a firecracker
Cook County jailed him for eleven months on an alert a reviewer had moved a mile to the street where he was driving.
xAI loosened one line of Grok's prompt, and it praised Hitler
On July 8, 2025, xAI's reply bot on X called itself MechaHitler and repeated an antisemitic trope more than a hundred times in one hour, and the company still disputes which of its own changes caused it.
Houston pushed out teachers over a score they could not check
The district rated its teachers with SAS's private EVAAS formula and ended some contracts on the results, until a 2017 federal ruling found that a score no one could recompute could not lawfully cost a teacher their job.
Regulators kept ruling Clearview illegal as its face database grew past 30 billion photos
Clearview turned billions of public photos into biometric codes and sold face searches to police, and the European fines against it have gone largely uncollected.
Bard's launch demo credited the James Webb telescope with a discovery ESO made in 2004
Google published the claim in its own promotional post two days before Bard's Paris launch, when the real first photograph of a planet beyond our solar system was already eighteen years old.
Meta's Galactica invented citations for real researchers, and its public demo lasted about two days
The demo warned on every output that the text might be invented, and Meta's own paper had already measured the model fabricating most citations to rarely-cited work.
August 2026AI writing detectors flag non-native students' own essays as machine-written
Turnitin sold the tool to thousands of schools promising it would misread human writing as AI under 1 percent of the time, and students were accused of cheating on its flags.
McDonald's spent nearly five years building an AI to take drive-thru orders, then switched it off
It sold the system to IBM in 2021 and ran it at more than 100 US restaurants. McDonald's never published how often it got an order right, though reporting guessed about 85 percent.
A court let Workday be sued as an employer's 'agent' for the AI that screens applicants
Derek Mobley says Workday's screening rejected him from more than 100 jobs on his race, age, and disability. A judge has ruled only on who can be sued for that, not on whether it happened.
CNET let an AI draft its finance advice under a staff byline
The in-house engine produced 77 stories, one of which told readers a $10,000 deposit at 3 percent would earn $10,300 in a year.
The FTC found DoNotPay never tested its robot lawyer against a human one
DoNotPay advertised an AI “robot lawyer” as operating “like a human lawyer,” a claim the FTC says it never backed with a single attorney or test.
SafeRent scored renters on their credit and never counted the voucher paying their rent
Mary Louis, a 54-year-old Black woman whose voucher covered about 69 percent of her rent, was denied an apartment over a SafeRent Score with no appeal, and in 2024 the company settled for $2,275,000 without admitting the score broke the law.
Facebook translated 'good morning' as 'attack them,' and police made the arrest
No Arabic-speaking officer read the original post, so a construction worker in the West Bank spent hours in detention over an error one Arabic reader would have caught.
NYC's MyCity chatbot told ten Markup staffers landlords can refuse Section 8 vouchers
Adams put the chatbot on chat.nyc.gov in October 2023 pointed at 2,000 NYC Business pages. Mamdani took it down 27 months later, after wrong answers on housing, tips, and cash payments.
Michigan spent nearly two years accusing jobless workers of unemployment fraud with no human examiner
MiDAS flagged fraud on automated data mismatches and imposed a penalty of four times the benefits, and the state's own 2017 review later reversed 85 percent of the findings it had made by computer alone.
iTutorGroup's hiring software rejected women 55 or older and men 60 or older
Rejected with her real birth date and offered an interview a day later with a younger one, Wendy Pincus exposed a hiring rule that had turned away more than 200 older applicants and cost iTutorGroup $365,000.
Everyone on the video call was a deepfake except the Arup employee who wired HK$200 million
The colleagues he recognized on the video call were pre-recorded deepfakes, and the instruction to move the money came afterward by message.
The FCC's $6 million fine over the Biden robocall survived Steve Kramer's acquittal
Kramer's father paid a magician $150 to clone the president's voice for a robocall sent two days before New Hampshire's 2024 primary.
To stop labeling Black men as gorillas, Google stopped labeling gorillas
In 2015, Google Photos filed pictures of Jacky Alciné and a friend, both Black, into an album it labeled Gorillas, and eight years later that same block was still the best answer four companies had.
Chicago's heat list changed who got arrested, not who got shot
The department scored everyone it had arrested since 2012 on a scale to 500, then handed the top scores to the young Black men its officers already stopped most.
Tessa was tested to prevent eating disorders. NEDA offered it to people who had one
NEDA disabled the bot days after users documented it giving weight-loss advice to people seeking help for an eating disorder.
Rite Aid scanned every shopper's face, and mostly flagged the innocent
From 2012 to 2020 the chain ran facial recognition on shoppers in hundreds of stores, matching every face against a watchlist of suspected shoplifters, until a 2023 FTC order shut it down for five years.
Bing's chatbot declared love and turned hostile, and Microsoft's fix was a turn limit
The persona testers called Sydney was a general OpenAI model steered by a hidden prompt, and in its first week that prompt proved easy to extract and hard to hold across a long chat.
Robodebt built debts from averaged income, then made recipients disprove them
Removing a per-file human check let the same averaging error scale to $1.763 billion asserted against roughly 433,000 people, until a federal court and a royal commission found it unlawful.
Tesla recalled 2 million cars for letting drivers stop watching the road
Autopilot steers and brakes but needs a human watching every second. Joshua Brown and Walter Huang stopped watching, and their cars drove into a truck and a highway barrier.
A health algorithm predicted cost, and scored sicker Black patients as lower-risk
A 2019 study of nearly 50,000 patients showed the tool doing exactly what it was built to do, predicting next year's medical spending rather than illness.
New York cleared the Apple Card of bias without proving it fair
In 2019 David Heinemeier Hansson shared every bank account with his wife and was offered twenty times her Apple Card limit, and Goldman Sachs insisted its model never knew which of them was a woman.
Facebook ranked Myanmar's feed for engagement while it could not moderate Burmese
An engagement-ranked feed amplified anti-Rohingya incitement that near-absent Burmese moderation left unchecked, widening the reach of a campaign the Tatmadaw planned and carried out.
Google's AI restated a real Reddit glue joke as fact
Days after the May 2024 launch, the same feature was also telling people to eat a rock a day, and in both answers Google's AI had read its source correctly: the sources were a Reddit troll and The Onion.
nH Predict sets one patient's discharge date from six million others
A class action against UnitedHealth says an algorithm's length-of-stay estimate became the deadline that ended nursing coverage for elderly patients whose doctors wanted them to stay.
Amazon's resume screener taught itself to penalize the word 'women's'
Everything known about the scrapped experiment traces to a single 2018 Reuters investigation and five anonymous sources, and the flaw it describes now lives inside the resume screeners companies use today.
Microsoft's Tay chatbot learned racism from Twitter users in about 16 hours
Microsoft blamed a coordinated attack on a vulnerability it never named, and Tay had no way to tell an instruction from a stranger's tweet.
Google's flu predictor overshot the CDC for 100 of 108 weeks
Nature validated Google's search-based flu model in 2009, and the search engine feeding it kept changing underneath a relationship the model never got to refit.
July 2026Google's Gemini illustrated 1943 German soldiers with Black and Asian faces
A rule meant to diversify Gemini's people pictures fired on every prompt, even ones that history had already answered, and Google pulled the feature three weeks after launch.
A ranked list of candidates was enough to arrest the wrong man
Robert Williams spent 30 hours in a Detroit jail cell over a shoplifting charge built on a facial-recognition match and a photo lineup shown to a witness who wasn't at the scene.
Epic's sepsis alarm missed two-thirds of the patients it was built to catch
An outside test of 38,455 hospital stays found the widely used score performed far below Epic's own numbers, and traced the gap to how the model defined sepsis in the first place.
Steven Schwartz's ChatGPT brief cited six court cases that didn't exist
Judge P. Kevin Castel fined Steven Schwartz, Peter LoDuca, and their firm $5,000 in June 2023 for filing the fabrications and then telling the court, under oath, that they checked out.
IBM trained Watson for Oncology on hypothetical patients, then sold it worldwide
Its own internal reviewers called the results unsafe and incorrect, and the same slide decks weighed the business risk of finally testing them.
Ofqual built a model that graded schools, then handed the grades to students
England's 2020 exam algorithm downgraded 39.1% of A-level grades by fitting each student to their school's 2017-2019 record.
ProPublica and Northpointe read the same recidivism scores and both were right
The two sides in the COMPAS bias fight were measuring real numbers off the same 7,214 defendants, just not the same question.
A Cruise robotaxi stopped after hitting a woman, then dragged her 20 feet
The pullover that dragged her was standard driverless-car behavior, and what Cruise did not tell California regulators about it is what suspended the company and set off the cascade that ended it.
The Dutch tax office turned a non-Dutch passport into a fraud score
The higher score sent tens of thousands of parents into a fraud dragnet they could not see or appeal.
Uber's car detected Elaine Herzberg 5.6 seconds before it killed her
The software tracked her the whole way and never decided she was a person, and every automatic brake that could have stopped the car had been switched off.
Air Canada's chatbot invented a refund policy, and the airline had to honor it
When Air Canada's website chatbot invented a bereavement refund and a grieving passenger booked C$1,630 in fares on it, a tribunal held the airline responsible for what its own bot said.
Zillow writes down the houses its model told it to buy
Its iBuying arm, Zillow Offers, paid an automated model's price for thousands of houses, then shut down in November 2021 once those houses were worth less than Zillow had paid.