Amazon's resume screener taught itself to penalize the word 'women's'

A resume engine that learned to prefer men

In 2018, Reuters reported that Amazon had quietly built, and then abandoned, a machine-learning tool that scored job applicants and had learned to penalize women.¹ The account is Jeffrey Dastin's investigation, sourced to five people who worked on the effort and spoke on condition of anonymity.¹ Every specific that follows traces to those five people.

By that reporting, a team began building the programs in 2014 to rate applicants from one to five stars, much as shoppers rate products on the site.¹ They created about 500 models, each tied to a particular job function and location and each trained to recognize some 50,000 terms pulled from past applicants' resumes.¹ The training set was ten years of resumes Amazon had received, most of them from men.¹ The system, Reuters wrote, "taught itself that male candidates were preferable."¹ It penalized resumes containing the word "women's," as in "women's chess club captain," and it downgraded graduates of two all-women's colleges the report did not name.¹

The fix carried no guarantee

Amazon's engineers edited the programs to make them neutral to those particular terms. That, Reuters reported, was no guarantee the models would not find other ways of sorting candidates that turned out to be discriminatory.¹ By the start of 2017, with executives having lost hope in the project, the team was disbanded.² The tool was reported in October 2018, a year and a half after it had already been shut down.¹

That timeline, and every figure in it, rests on one origin. The Reuters story was carried word for word by wire subscribers, including the Irish Times and RTÉ, but a reprint of the same report is not a second source; each is Dastin's article again.² Amazon itself declined to discuss the engine with Reuters.¹ When other outlets asked, the company said the tool "was never used by Amazon recruiters to evaluate candidates" and had been "only ever used in a trial and developmental phase."³ That sits against what Reuters' sources described: recruiters who "looked at the recommendations generated by the tool ... but never relied solely on those rankings."¹ Both accounts agree on the point that bounds the harm. The ratings were never the sole basis for a hire, and the tool never became a live screen. Nothing in the record says it rejected real applicants at scale.

Why deleting the word did not delete the bias

A model trained to reproduce a decade of Amazon's hiring reproduced its skew toward men.¹ The training data recorded whom Amazon had hired, and Amazon had mostly hired men. A model built to predict which resumes resemble past hires will, when past hiring favored men, learn to favor the signals that came with being a man.

So the engineers deleted the obvious signal and made the models ignore the word "women's."¹ It did not fix the bias, because being a woman was written into the data in more than one place. A women's college, a sport, a verb more common on one group's resumes than another's: any of these can stand in for the field that was removed. A feature that carries the information of a protected trait without naming it is a proxy. The formal name for the failure is redundant encoding, the trait encoded redundantly across correlated features, so a model can rebuild it after the direct field is gone.⁴

This is not particular to Amazon. Solon Barocas and Andrew Selbst named redundant encoding as the core obstacle to fair machine learning in employment. Membership in a protected class is often "encoded in other data," they wrote, so scrubbing the explicit field leaves the model able to reconstruct it.⁴ Cynthia Dwork and colleagues described the same move in formal terms. They called the strategy of dropping protected attributes "fairness through blindness," and showed it can be beaten by a test built from correlated data that is "in practice, essentially equivalent."⁵

No one has to intend this. Take a screener that is never shown an applicant's gender (illustrative, not something Amazon's tool is known to have done). Say the resumes it learns from show that people who list a certain hobby, or a certain two-year gap, tend to match past hires. The model will reward or punish that signal, and through it the trait behind the signal. No field is labeled for the trait, and the people reading the output would see only a ranking.

This is also why the last line of defense does not hold. Recruiters at Amazon looked at the tool's ranking, by the sources' own account.¹ A biased score that a person then approves, or merely starts from, has still shaped the decision. Deleting the variable does not delete the bias, and adding a human does not launder it.

The same weakness in today's screeners

Amazon's tool never went live. The failure behind it did. Automated resume screening is common enough now that regulators have written rules for it. The EEOC treats an algorithm that rates applicants as a "selection procedure" under Title VII. It applies the four-fifths rule: if one group's selection rate is less than 80% of the highest group's rate, that is a rule-of-thumb sign of adverse impact.⁶ The agency stresses it is only a rule of thumb, and that an employer stays liable even when a vendor built the tool.⁶ In the EEOC's own worked example, if 80 white and 40 Black applicants apply and 48 and 12 advance, the selection rates are 60% and 30%; their ratio, 50%, sits below the 80% line.⁶

New York City went further. Local Law 144 has been in effect since January 2023 and enforced since that July. It bars employers in the city from using an automated employment decision tool unless an independent bias audit has first measured the tool's selection rates and impact ratios across sex and race categories. The audit results must be published.⁷ Whether the audit changes anything is a separate question: a companion lesson on how that law works in practice found that employers largely decide for themselves whether they are even covered.

And the bias Amazon's engineers saw by 2015 is measurable in current tools. In 2024, University of Washington researchers tested three open-source language models on more than 550 real resumes. They swapped in 120 names associated with white and Black men and women across nine occupations, more than three million comparisons in all.⁸ The models favored white-associated names 85% of the time and Black-associated names 9%; male-associated names 52% and female-associated names 11%. Black-male-associated names were never preferred over white-male-associated ones.⁸