'Human parity' in speech recognition came down to how you count the humans

Parity arrives as a percentage

In October 2016, a team at Microsoft announced that its speech recognition system had reached "human parity" on conversational telephone speech. On the Switchboard portion of a standard test set, the system missed 5.8 percent of the words, while a professional transcription vendor given the same audio missed 5.9 percent.¹ By the field's own yardstick, a machine now transcribed casual conversation about as accurately as a person.

That yardstick is Word Error Rate, and the 5.8 percent is not a reading off a thermometer. It is a count. To produce it, you line the system's transcript against a reference a human prepared. Then you tally the word edits it would take to turn one into the other, and divide by the number of words in the reference. The percentage is edits over reference words, and nothing else. Everything the headline meant rests on two choices: which edits get counted, and what reference they are counted against.

How a transcript becomes a number

Begin with the two lines of text. The reference is what was actually said. The hypothesis is what the system wrote down. Scoring lines them up word for word, picking the alignment that needs the fewest edits, and sorts every mismatch into one of three kinds. A substitution is one word standing in for another, as when the system writes "slow" where the speaker said "quick." A deletion is a reference word the system left out. An insertion is a word the system added that no one spoke. Correct words cost nothing. The reference-scoring tool the field settled on, NIST's sclite, finds that alignment by minimizing a weighted distance. It charges 4 for a substitution, 3 for a deletion, 3 for an insertion, and 0 for a correct word, and keeps whichever reading of the two lines is cheapest.²

Written out, the score is the three edit counts added together, over the length of the reference:

Take a six-word reference and a system that got most of it:

The split into the three kinds is less obvious than the two lines make it look. The same 50 percent could be read as three substitutions rather than one substitution, one deletion, and one insertion, and the total error would come out identical. sclite's weights are what make the intuitive reading the official one. One substitution, one deletion, and one insertion cost 4 plus 3 plus 3, or 10, while three substitutions cost 12, so the alignment that matches a person's intuition is also the cheapest.²

The percentage also has no ceiling at 100. Because insertions are counted, a system that pads its output can score worse than getting everything wrong. Transcribe the single word "hello" as "hello there my friend" and you have three insertions over a one-word reference, a Word Error Rate of 300 percent.³

What the count can settle

Held to one test set, one reference, and one set of text-formatting rules, Word Error Rate does the job it was built for. If two systems transcribe the same audio against the same reference, the one with the lower rate genuinely made fewer word errors, and the distance between them is real. It is why speech-recognition leaderboards are Word Error Rate tables, and why the Microsoft and IBM systems could be compared at all: they ran on the same NIST tapes.

The score is built on raw string comparison, so it charges the system for differences that mean nothing to a listener: "okay" where the reference has "OK," "7" for "seven," a comma the reference omits. When OpenAI built Whisper, a speech model this course has already met, its team wrote a text normalizer to strip those non-semantic differences before scoring, and reported that on some datasets the step alone cut the measured error rate by as much as half.⁴ None of the words the system got right had changed. Only the bookkeeping had. A Word Error Rate is comparable only to another figure normalized the same way. That is the first question worth asking of any two scores placed side by side.

The human baseline moves

Return to that 5.9 percent human figure. It is the softest number in the story. Microsoft measured it by slipping the test audio, unlabeled, into a commercial vendor's ordinary weekly workload: one transcription pass from scratch, one pass correcting it, nothing more elaborate.¹ On the harder CallHome portion, where friends and family talk loosely rather than strangers on an assigned topic, that same vendor scored 11.3 percent against the system's 11.0. Parity seemed to hold on both halves.¹

Microsoft ran an error analysis and found that its system and the human transcribers stumbled in the same places, on the same short function words, with the same difficult speakers. Judges shown the two transcripts could not reliably tell which was the machine's.¹ Where the system erred, a person often erred too, and not in ways that wrecked the meaning any more than the human's did. On this test set, the number was not quietly hiding a difference in kind between the two.

What it was hiding was the softness of the thing it measured against. Prompted by the gap between Microsoft's 5.9 percent and an older, widely repeated 4 percent figure, IBM commissioned its own measurement of the same tapes, using three independent transcribers whose work a fourth senior transcriber checked. After that check, the best transcriber reached 5.1 percent on Switchboard and a "surprisingly low" 6.8 percent on CallHome, against Microsoft's 11.3.⁵ The human error rate on the hard set had nearly halved, though no one had spoken differently. The transcript was simply prepared with more care, and IBM traced almost the whole difference to deletions: the rushed vendor pass dropped words the careful one kept.⁵

Reviewing the dispute in its 2017 follow-up, Microsoft conceded the point in plain words.

Two readings survive, and both are true. On Switchboard, parity was real: Microsoft's improved 2017 system reached 5.1 percent, matching IBM's best human transcriber exactly.⁶ But Switchboard is an easy test, with 36 of its 40 evaluation speakers also present in the data the systems trained on. On CallHome, where the human number had moved to 6.8 percent, no machine had reached parity at all.⁵ The milestone was genuine, as the linguist Mark Liberman noted at the time.⁷ It was also a milestone on one benchmark, against a human baseline whose value depended on how hard someone had been asked to try.

The errors a percentage can't tell apart

The parity result held up because its errors happened to be benign. That is not something Word Error Rate can promise, because the metric cannot see meaning at all. It counts word edits, and every edit costs the same. Dropping the word "not" from "do not resuscitate" is a single deletion, worth exactly what swapping a filler "uh" for an "um" is worth. A fluent, confident sentence the speaker never said scores the same as an audible mishearing of equal length. The number reports how many words differ, never which of the differences matters.

Reported over a whole test set, that blindness compounds. A headline rate is an average, so a few segments where the system invented text can hide inside a low overall number, smoothed away by the clean segments around it. Whisper does exactly this. OpenAI's own analysis records it getting stuck in repeat loops, skipping the first or last words of a segment, and at times producing "a transcript entirely unrelated to the actual audio."⁴ Someone reading the caption has no way to tell, from the score alone, whether an error was a harmless slip or a sentence of confident fiction.

The number travels badly, too. Run the same systems on accented or dialect speech and it moves sharply. A 2020 study of five commercial systems found an average Word Error Rate of 0.35 for Black speakers against 0.19 for white speakers on matched conversational audio, close to double in every system tested. The authors traced the gap to acoustic models trained on too little Black speech.⁸ Add background noise and even the ranking can flip: systems that post nearly identical numbers on clean audio pull apart as the signal degrades.⁴ Microsoft's 5.8 percent was true. It was true of one corpus, read against one reference, normalized one way.