DQN reached ‘human-level’ Atari on 29 of the 49 games

One network learned 49 Atari games from raw pixels

In February 2015, a team at Google DeepMind published “Human-level control through deep reinforcement learning” in Nature. A single convolutional neural network, with one fixed set of hyperparameters, learned to play 49 different Atari 2600 games.¹ Its only inputs were the raw screen pixels and the running game score. The paper reported that it “performed at a level that was comparable to that of a professional human games tester across the set of 49 games.”¹ That sentence, compressed to two words, is why the paper is famous.

Those two words, “human-level,” hide a specific and checkable meaning.

Inside one Breakout paddle decision

Picture the Breakout screen: a paddle along the bottom, a ball falling toward it, a wall of bricks up top. The network never sees “a ball” or “a paddle.” What it sees is four grayscale snapshots of the screen, each shrunk to 84 pixels square and stacked into one block of 84 by 84 by 4 numbers.¹ Four frames rather than one, because a single still cannot tell you which way the ball is heading.

Now run the decision the network runs. Feed that block in. Out comes one number for each thing the joystick can do: move the paddle left, move it right, hold still. Each number is the network’s estimate of the total future score if you take that action now and keep playing afterward. Pick the action with the largest number. That is the entire decision.

That estimate has a name. It is Q(s,a), the value of taking action a in state s, defined as the total reward you can expect from that point on if you keep playing well. An algorithm called Q-learning, first described in 1989, learns this value function from experience. It converges toward the best possible values however clumsily the agent played while learning. The agent then acts greedily, taking the highest-valued action at each step.² Learning a value and acting on it, with no model of the game and no search over future moves, is what makes this a value-based method.

The network learns Q by correcting itself. After each action, it has a better estimate of what the earlier state was worth: the reward just collected, plus the largest Q-value available in the state it landed in. It nudges its weights, by gradient descent, to pull the earlier estimate toward that updated one.² This is bootstrapping: the target the network trains toward is built partly from the network’s own current guess about the next state.

Two tricks stop the training from diverging

Build the plain version anyone would try. Take a neural network, show it the stacked frames, have it predict a Q-value for each action, and train it by gradient descent on that bootstrapped target as the agent plays. Do that and it does not learn; it diverges. Two separate problems wreck it, and each of the paper’s two tricks removes one.

The first problem is that consecutive frames are nearly identical. As the agent plays, each frame it trains on looks almost exactly like the one just before, so the network learns from a stream of near-duplicates and forgets whatever it saw a few seconds earlier. It is like studying for an exam by reading one paragraph ten thousand times and never turning the page. The fix is experience replay. Store each transition the agent lives through, its state, action, reward, and next state, in a memory of the last million. Then train on a random batch drawn from that memory rather than on the latest frame. Drawing at random “breaks these correlations and therefore reduces the variance of the updates.”³ Experience replay was already present in the 2013 precursor to this work, a seven-game version the same team posted two years earlier.³

The second problem is subtler, and fixing it is what the 2015 paper added. The target the network trains toward is computed from the network’s own weights, the very weights each update changes. Every time the network shifts toward the target, the target itself moves, because it was built from the weights that just moved. The network is chasing a number it is simultaneously pushing away. The fix is a separate target network. Keep a second, frozen copy of the network and compute the training targets from that copy. In the paper’s words, “every C updates we clone the network Q to obtain a target network” and use it to generate targets for the following C updates.¹ The 2013 version had no such frozen copy; its targets came from the previous iteration’s weights.³

How much do the two tricks matter? The authors turned each one off and measured. On Breakout, removing experience replay alone collapsed the score about thirtyfold, and removing both dropped it to the floor of random play.

Breakout is not special. Every one of the five games the authors tested collapsed the same way when both tricks came off.¹ This is the paper’s durable contribution: not that a network reached some particular score, but that these two changes made “a neural network trained by Q-learning” stop diverging and start working at all.

What the ‘human-level’ bar actually counted

“Human-level” was not a judgment call. It was a formula. The paper set the top of each game’s scale to a professional human games tester’s score and the bottom to random play. It called the agent “human-level” on a game once the agent reached at least 75% of the way from random up to that tester.¹ The threshold is written down plainly in the paper: performance “operationalized as a level of 75% or above.”¹ On that 75% bar, DQN cleared 29 of the 49 games.¹

Two facts sit underneath that number. The first is the human it was measured against: one professional games tester, playing each game for about twenty sessions of up to five minutes each, with no pausing, saving, or reloading.¹ Every “human-level” claim in the paper rests on that one person’s afternoon. The second is the height of the bar. It sits at 75%, not at the human’s actual score. Raise it to the full human score, matched or beaten, and the count drops to 23 of 49. NBC News reported exactly that figure the day the paper appeared: “The AI outscored humans on 23 out of 49 games.”⁴ The six games that sit between the 75% line and the human’s real score are where the phrase “human-level” is doing its quiet work.

The bottom of the scale is the sharpest case. On Montezuma’s Revenge, DQN scored 0. The tester scored 4,367 and random play scores 0, so DQN’s 0 lands at 0.0% of human, the lowest of all 49 games.¹ The reason is built into value learning. Q-learning can only raise the value of an action that led, somewhere down the line, to a reward. Montezuma’s Revenge pays its first point only after a long, exact sequence: climb down a ladder, cross a floor, collect a key, backtrack to a door. An agent flailing at random almost never completes that sequence by chance, so it never once sees the reward, and an action that is never rewarded is never reinforced.

The template spread faster than the agent improved

Two claims travel under “human-level control,” and the decade since has split them. The first, that a deep network trained by reinforcement learning could work at all, held and spread: the search-and-self-play systems behind AlphaGo and the reasoning models of 2025 are its descendants. The second, that this particular agent was general or robust, did not survive contact.

Each of those 49 results came from 50 million frames of play per game, about 38 days of game experience.¹ The agent needed weeks of play on a single game to reach a score the tester reached in an afternoon.

Robustness has fared worse. Deep reinforcement learning as a field turned out to be fragile. In 2018, Henderson and colleagues ran several deep-RL algorithms with the same settings, changing only the random seed. Ten identical runs, split into two sets of five, gave averages that differed at t = -9.09, p = 0.0016.⁵ Their tests were policy-gradient methods on simulated robot control, not DQN and not Atari, so this is not a measurement of DQN’s own runs. It is evidence about the wider field DQN founded.

Montezuma’s Revenge stayed hard for years. Reviewing the lineage in 2020, MIT Technology Review noted that agents still failed on the games that demand exploration, and that even the system which finally beat all 57 could not play them without retraining for each.⁶ DeepMind’s own retrospective is blunter. For years, it writes, “all deep reinforcement learning agents have consistently failed to score” on four games, Montezuma’s Revenge among them. Only in 2020 did one clear the human baseline across the full set.⁷

“Human-level control” gets cited today as the moment machines reached human competence at Atari. The scores span an enormous range, from 0% of human on one game to many multiples on others, all measured against one tester on a bar set at 75%. The phrase “human-level” describes the middle of that spread and neither of its ends.