Employers decide whether New York City's hiring-bias audit applies to them

Coverage starts with the employer's own judgment

An employer in New York City may not use an automated hiring tool to screen a candidate until an independent bias audit of that tool has been posted on the company website, refreshed within the year, and the employer has given candidates at least ten business days' notice and a chance to ask for an alternative.¹ That rule has bound since 5 July 2023. The decision that governs everything before it, though, belongs to the employer. A tool is a covered "automated employment decision tool" only when its output is "used to substantially assist or replace discretionary decision making," and the law never quantifies "substantially."¹ The employer reads its own tool against that line first, before any auditor is retained, and the enforcing agency endorses the sequence: the Department of Consumer and Worker Protection instructs employers to assess their own tools against the threshold.²

So the safeguard is real and its first switch sits with the party it is meant to check. The audit the law demands is a genuine statistical test, and a few employers have run and posted one. The compliance record is close to empty all the same. A mandate reaches only as far as the definition that admits a tool to it, and here the regulated party reads that definition before anyone else does.

A screener that learned to skip women

The failure the audit answers is not hypothetical, though its best-known case is secondhand. Amazon built an experimental system, beginning in 2014, that scored job applicants from one to five stars. Trained on a decade of past résumés from a mostly male engineering workforce, it reportedly taught itself to downgrade résumés that contained the word "women's" and the names of certain all-women's colleges, and the company scrapped it before it was ever the sole basis for a hiring decision.³ The reporting is Jeffrey Dastin's for Reuters; the details here come from MIT Technology Review's account of it. It is a reported episode, not a proven exhibit.

A hiring model needs no one's intent to discriminate in order to produce a disparity. It learns the patterns in its training data, and if past hiring skewed, the pattern skews with it. Federal law already reaches the result: the Equal Employment Opportunity Commission has said an employer can be liable for a tool's discriminatory effect even when an outside vendor built and ran it.⁴ Local Law 144's premise is narrower and procedural. The disparity, if it is there, should be measured and shown before the tool sorts anyone.

One selection rate divided by another

What the law requires is a single statistic, computed group by group. Start with the selection rate: the share of a group's applicants who are picked, whether hired, promoted, or advanced.⁴ Federal enforcement has long used a benchmark called the four-fifths rule. If one group is selected at less than four-fifths, or 80 percent, of the rate of the most-selected group, that gap is generally treated as evidence of adverse impact.⁴ New York turns that benchmark into the number an auditor must report, the impact ratio: one group's selection rate divided by the selection rate of the most-selected group.⁵

Work it once. If women are selected at 45 percent and men, the highest group, at 60 percent, the impact ratio is 45 divided by 60, or 0.75, which sits below the 0.80 line and is flagged.⁴⁵ The audit computes that ratio for sex, for race and ethnicity across the federal reporting categories, and for the two combined, the intersectional cells the final rule added.⁵ A real one exists to read. The audit of the vendor HireVue's tool, run by the consulting firm DCI, reports exactly this ratio across roughly a dozen scored competencies; because the tool sorts applicants into tiers rather than a yes or no, the ratio is taken at the threshold between tiers, and almost every cell clears 0.80.⁶

What a passing audit does not settle

A passing audit certifies less than it appears to. It says that, on the historical data the auditor was handed, top-tier selection rates stayed within four-fifths across the groups that were reported. It does not say the tool is unbiased, and it does not say the tool selects the best candidates. Those are separate claims the number never reaches.⁶ The HireVue report shows the gap in its own tables. Every row for Native Hawaiian or Other Pacific Islander applicants and for Native American or Alaska Native applicants reads zero applicants and reports no ratio at all: whole protected groups drop out of the test for want of data.⁶ The auditor is candid about the method. In a footnote, DCI writes that several of Local Law 144's requirements "are not aligned to contemporary adverse impact analysis practices," and that it ran them only to meet the law.⁶

The independence the statute promises is thin in the same way. The law asks for "an impartial evaluation by an independent auditor" and stops there; it does not bar that auditor from being chosen and paid by the employer whose tool is under review.¹ Analysts writing at Tech Policy Press warned early that the independent auditor "could be any number of consultants or technology firms," and that the arrangement risks "creating a cottage industry around rubber stamping AI systems."⁷ An audit paid for by its subject can be honest, but nothing in the rule requires it to be adversarial.

Eighteen audits and two complaints

In a 2024 study called "Null Compliance," a research team sent 155 student investigators to examine 391 employers for signs of compliance. Across all 391, 18 had posted a bias-audit report and 13 had posted the required candidate notice.⁸ Among the 267 that had active New York City job listings at the time, 14 had posted an audit and 12 a notice, roughly one covered-looking employer in twenty.⁸

The authors are careful about what the count means. They cannot know how many of the 391 were actually covered, because coverage turns on each employer's own "substantially assist" judgment. An employer "can off-ramp from the regulatory decision tree," they write, "by claiming (correctly or incorrectly) that their decision-making process does not 'substantially' rely upon the outputs of the AEDT."⁸ They name the resulting state null compliance: a condition in which missing evidence of compliance cannot be read as non-compliance, because the investigator cannot tell whether the tool was ever in scope.⁸ A low count is consistent with widespread evasion, and equally with many employers rightly concluding their tools are exempt. What the study establishes is that from outside no one can tell which.

Enforcement does little to resolve the ambiguity. Local Law 144 is enforced only on complaint, and the penalties are modest: up to $500 for a first violation and up to $1,500 for each later one, with every day a non-compliant tool runs counting as a fresh violation.¹ In December 2025 the New York State Comptroller reported that the Department of Consumer and Worker Protection had received just two complaints about automated hiring tools within the audit's scope, and found the agency's complaint process "ineffective" at even routing them.⁹ The federal backstop thinned in the same window. The EEOC removed its 2023 guidance on AI and adverse impact in January 2025, though Title VII itself still applies.¹⁰⁴

A safeguard the regulated party gets to scope

Local Law 144 did switch something on. An employer that concedes its tool is covered now owes a public statistical test that did not exist before 2023, and a handful have posted one. What the law cannot switch on by itself is its own coverage, and the same narrowing move appears at two scales. Inside a posted audit, the rule lets the auditor drop any group that makes up less than 2 percent of the data, and the "Null Compliance" authors found employers omitting a fifth of their impact ratios on that ground.⁵⁸ Outside the audit, the "substantially assist" definition lets an employer decide the whole tool is uncovered and post nothing. In both, the party being measured sets what gets measured.

There is a family resemblance to the European Union's compute threshold for frontier models, where a regulator leans on a number it controls to decide which systems face scrutiny; here the test for a risky tool is instead the regulated employer's own reading of a word the statute declines to define. The impact-ratio math is sound, and the audits that do get posted are real. The mandate's reach is fixed one step earlier, by whoever decides a tool is in scope, and in New York that decision starts with the employer.¹