The 2016 WaveNet paper never reports its own generation speed

One hundred sentences, eight listeners, one score

In September 2016, Aäron van den Oord and a team at DeepMind posted "WaveNet: A Generative Model for Raw Audio" to arXiv.¹ It became the paper people cite when they say AI-generated speech got good enough to sound human. What it actually reports is one experiment: a single listening test, run separately in North American English and Mandarin. Each of 100 held-out sentences from a lone professional speaker, rendered by every system under test, was rated by eight crowdsourced listeners on a five-point scale, after ratings that failed a headphone check (about 40 percent of them) were thrown out.¹

WaveNet's synthesized English averaged 4.21 out of 5. A natural recording of the same sentences averaged 4.55 in the same test.¹ The best prior systems scored lower still: 3.67 for a statistical parametric model, 3.86 for a system that stitches speech together from recorded fragments.¹ The table below is the paper's own Table 1.

DeepMind's own announcement of the result stuck to the same scale: it framed WaveNet as "reducing the gap with human performance by over 50%," not as a system that had closed it.² No claim of matching human speech, generating it in real time, or shipping it in a product appears in either the paper or that announcement. Why real-time generation had to wait for a different paper starts with what WaveNet actually predicts, one sample at a time.

Guessing 16,000 samples a second

A digitized recording of speech is a long list of numbers, one for every instant a microphone was sampled. An ordinary recording samples at least 16,000 of them every second.¹ WaveNet generates that list one number at a time: to produce the next sample, the network looks at every sample it has generated so far and outputs a probability distribution over what comes next, a design known as autoregressive. Picture it on three numbers instead of 16,000: given a run like −0.2, 0.1, 0.3, the model's whole job is to guess a fourth. WaveNet does that guess 16,000 times over for a single second of audio, each guess built on everything guessed before it.

Reaching back far enough in that list to shape a believable sample is exactly what makes raw audio hard. A plain convolutional layer only looks a handful of samples back, so seeing far enough into the past to shape a single natural-sounding sound would take either an enormous filter or dozens of stacked layers.¹ WaveNet's stack instead skips over most of what lies in between. The paper's own diagram shows four layers whose skip widens by doubling at each one — 1, 2, 4, then 8 samples. A unit at the top layer has effectively looked back across all of them, while doing no more work than an ordinary convolution spends looking back across four.¹ The paper calls this a dilated causal convolution: causal because a sample never looks ahead of the one it's predicting, dilated because of the widening skip.¹

Even stacked this way, the paper reports that its unconditional model can only draw on about 300 milliseconds of its own history, roughly the length of two or three phonemes, before that memory runs out.¹ Left to generate speech with no text to steady it, the model invents words that sound like language, then loses the thread and starts inventing a new one once it drifts past that 300-millisecond reach.¹

Built this way, the model was trained on three kinds of audio: 24.6 hours from one English speaker and 34.8 hours from one Mandarin speaker for text-to-speech. Music used roughly 200 hours of tagged clips plus 60 hours of scraped piano recordings, with no listening-test score reported for either.¹ Retrained as a plain classifier instead of a generator, the same design also set what the paper calls the best score yet for a model trained directly on raw audio on TIMIT, a standard speech-recognition benchmark. The result, an 18.8 phoneme error rate, is reported without a specific prior score to measure it against or a stated number of training hours.¹

The paper's only sentence about speed

On how long any of this takes to generate, the 2016 paper says exactly one thing: "the predictions are sequential: after each sample is predicted, it is fed back into the network to predict the next sample."¹ No timing figure, no real-time comparison, no statement about whether the method could run in a live product appears anywhere else in the text.

Those numbers arrived a year later, in a follow-up paper written by several of the same authors plus new co-authors. Running the original, undistilled network on a single Nvidia P100 GPU, DeepMind's own optimized implementation reached 172 timesteps a second.³ A live product generating 24 kHz audio needs 24,000 samples every second, so the original network ran at under one two-hundredth of that speed.³ The paper says plainly why that mattered: relying on sequential generation of one sample at a time made the network "poorly suited to today's massively parallel computers, and therefore hard to deploy in a real-time production setting."³

The fix, called probability density distillation, trains a second, feed-forward network to copy a pretrained WaveNet, closely enough that the new network can produce every sample of an utterance together instead of one after another.³ DeepMind's own description put it more plainly: the new network learns to generate "the first and last word — and everything in between — at the same time."⁴ On the same P100 GPU, the distilled network reached over 500,000 timesteps a second, which the paper calls three orders of magnitude faster than the network it copied.³ That distilled version, not the one described in the 2016 paper, is what DeepMind put into a live product the following year.

The name outlived the architecture

The distilled system launched inside Google Assistant in October 2017, the first product to run on what DeepMind called Google's newest TPU cloud infrastructure.⁵ DeepMind's launch post reported a new listening test with its own scores: 4.347 and 4.314 for two US English voices, 3.966 for a third voice, and 4.236 for a Japanese voice. The human baseline in that same test was 4.667.⁵

That 4.667 is not the 4.55 the 2016 paper measured for natural English speech, or the 4.21 it measured for natural Mandarin.¹ DeepMind repeated the same 4.667 figure again two months later, in a separate post about the underlying method.⁴ Three tests, three panels, three numbers for what a human voice scores on this same five-point scale. That is why a raw MOS travels badly between papers, and why WaveNet's own English score, 4.21, is the safer number to repeat than any percentage computed from it.

The 4.41 that gets quoted as WaveNet reaching near-human quality comes from the 2017 paper's own Table 1, but on a new 24 kHz recording, not the 16 kHz corpus the 2016 paper's 4.21 came from.³ On that new corpus, the original architecture and its distilled copy score identically, 4.41 each, beating a concatenative baseline retrained on the same higher-fidelity recordings at 4.19.³ That is a real result. It is not a like-for-like improvement over the number the 2016 paper reported: part of the gain is a better recording, not only a better model, and the 2017 paper does not separate how much of it is which.

A second deployment, Google's Cloud Text-to-Speech product, reported a fourth number five months later: an average score of 4.1, described as reducing the gap with human speech by "over 70%."⁶ That 70 percent is not the 51 percent the 2016 paper reported for English or the 69 percent it reported for Mandarin,¹ and Google's post does not say which language or which point values it is computed from.⁶ One paper's own two languages account for two of these percentages. A different product, two years later, supplies the third. Treating any one of them as "the gap WaveNet closed" would flatten a real difference between three separate measurements.

A 2021 academic survey of speech-synthesis methods, independent of DeepMind, credits WaveNet as "the first neural-based vocoder" while stating plainly that "although WaveNet achieves good voice quality, it suffers from slow inference speed."⁷ The vocoder families that took over production speech synthesis after it, built on generative adversarial networks and on diffusion, generate every sample of an utterance in parallel from the start.⁷ Parallel WaveNet reached that same output speed a different way, by distilling a slow teacher into a fast student.³ What survived from the 2016 paper is the idea underneath all of them: model the raw waveform directly, sample by sample, instead of hand-engineered vocoder features. What did not survive, on its own, is the architecture that first proved the idea worked.

Coverage of the 2016 announcement was not automatically looser than the paper. Engadget's write-up at the time reported the gap-closing result without claiming parity, noting that "we're still far from using a WaveNet-powered app."⁸ Exactly where the looser "human-level" shorthand entered the record afterward is not something any single article traces back to; it reads like a number drifting into a vaguer claim over years of retelling, not one overclaim anyone can point to.