Amazon's Rekognition falsely matched 28 members of Congress to arrest photos

The ACLU ran all 535 members at the default setting

In July 2018 the ACLU of Northern California took the official photo of every one of the 535 members of Congress and ran each through Amazon Rekognition, Amazon's commercial face-matching service, sold through AWS and pitched to police departments.¹ Face matching takes a probe, the face you are searching for, and compares it against a gallery, a stored set of faces. It returns candidates, each with a similarity score: a percentage for how closely the two faces resemble each other. The ACLU's gallery was 25,000 publicly available arrest photos.¹

Run at the service's default settings, Rekognition returned 28 matches. All 28 were false matches, cases where the system links photos of two different people: no member of Congress was the person in the arrest photo it was paired with. Nearly 40% of the false matches were people of color, though people of color make up about 20% of Congress. Six were members of the Congressional Black Caucus, among them the late Rep. John Lewis of Georgia. The whole scan cost $12.33.¹

Behind a match is a similarity score and a cutoff

A match is not something the system discovers. It is a similarity score that clears a cutoff the operator sets, called the confidence threshold. Below the line the candidate is dropped; at or above it, the system calls it a match. The ACLU used Rekognition's default threshold of 80%.²

Amazon disputed the test the next day, July 27, 2018. Dr. Matt Wood, who ran deep learning and AI at AWS, called the 80% default good for general use, such as identifying celebrities or spotting look-alike family members in a photos app. But it is "not the right setting for public safety use cases," he wrote, and for those Amazon recommends 99%. Amazon said it re-ran a similar test at 99% and got zero misidentifications against a corpus 30 times larger than the ACLU's.²

The ACLU's reply, the same week, made two points. A default is what an operator gets out of the box, and Amazon neither requires 99% nor gates the API on it. And Amazon's own recommended figure kept moving: its June 2017 walkthrough with the Washington County Sheriff's Office had set the threshold at 85%, and in the days around the test the recommendation went to 95% and then to 99%.³

That raising the threshold lowers the number of false matches is not in dispute. What the ACLU disputed is who has to know to raise it, and whether a higher cutoff fixes the whole problem. It does not, because the cutoff is only one of two things working against the operator.

A bigger gallery buys more chances to be wrong

The second thing is the size of the gallery. A one-to-many search compares the probe against every face stored, so running 535 probe faces against 25,000 arrest photos is more than thirteen million one-to-one comparisons.¹ Each comparison can go wrong on its own, and the more you run, the more chances a wrong one has to clear the line. This is the base-rate problem the ShotSpotter lesson works through: a test that is usually right still turns up false alarms once you run it enough times. A search of thousands of faces is exactly that many times.

So a false match does not require a bad algorithm. It requires a large gallery and a cutoff loose enough to let the closest wrong face through, and the ACLU supplied both. Amazon's retest points the same way from the other side. When it pushed the cutoff to 99% and searched an even larger set, its reported misidentifications fell to zero.² The threshold and the gallery are the two dials. The ACLU left the threshold at its factory setting, and the gallery of 25,000 gave the wrong faces their chances.

The wrong hits did not fall evenly

The 28 false matches skewed by race, but 28 is a small set, and that skew describes these particular matches rather than a measured error rate for any group.¹ To ask whether face systems really err more on some faces than others, you have to look at controlled audits, and the two most often cited measure a different task than the ACLU test.

Gender Shades, published in 2018 by Joy Buolamwini and Timnit Gebru, tested three commercial gender-classification systems, ones that guess a binary gender label from a face, from Microsoft, Face++, and IBM. Amazon was not among them. Darker-skinned women were misclassified at rates up to 34.7%, against a maximum of 0.8% for lighter-skinned men.⁴ The follow-up, Actionable Auditing by Inioluwa Deborah Raji and Buolamwini, added Amazon: on the same benchmark, Rekognition mislabeled the gender of darker-skinned women 31.37% of the time and of lighter-skinned men 0.0%.⁵ Both figures measure gender classification, not face matching. Guessing a face's gender is a different task from deciding whether two faces belong to the same person. A gap measured on one task, like the skin-tone gap in image labeling, does not automatically carry to the other.

Amazon drew exactly that line. In January 2019 Wood argued that facial analysis, finding generic attributes like gender, and facial recognition, matching a face to an identity, are "completely different in terms of the underlying technology," and that "using facial analysis to do facial recognition is an inaccurate and unadvised way to identify unique individuals." Amazon said its own gender-classification test on more than 12,000 images found "no significant difference in accuracy" across ethnicities, and that Rekognition searched MegaFace's one million images with zero false positives at the 99% threshold.⁶

So does matching itself err unevenly? The one study that measured demographic differences in face recognition rather than classification is NIST's, published in December 2019. Testing one-to-one and one-to-many algorithms from many developers on operational photo sets, NIST found false-positive rates that vary across demographic groups by factors of 10 to beyond 100. On U.S. mugshots, the one-to-many case that matches the ACLU's scenario, false positives were highest for American Indians, with elevated rates for African American and Asian faces. They were also higher for women than for men.⁷ That is a measured property of face matching. Raising the threshold lowers the overall false-match rate, which is Amazon's point, but does not by itself even out the gap between groups, which is NIST's.⁷ NIST never tested Rekognition itself, though, because Amazon did not submit it.⁷

Where face matching runs after the pause

On June 10, 2020, Amazon changed course. It announced a one-year moratorium on police use of Rekognition and asked Congress to write rules for the technology. It kept access open for Thorn, the International Center for Missing and Exploited Children, and Marinus Analytics, for anti-trafficking and missing-children work.⁸ It came amid the protests after George Floyd's death and one day after IBM left general-purpose face recognition. MIT Technology Review, which tracked the campaign from the Congress test through the audits, reported that the pause was narrow: time-limited, and silent on federal agencies such as ICE and DHS.⁹

The mechanisms did not pause with the product. Face matching still runs in policing and in identity checks, and the face-search vendors regulators have chased sell it against galleries far larger than 25,000. A match is still a similarity score past a cutoff someone chooses, a search still runs a probe against a gallery big enough to make some matches wrong, and where the errors have been measured, they still do not fall evenly. The next face search that returns a name is running the same arithmetic the ACLU paid $12.33 to expose.