The winning ticket turns up only after you train the whole network

The small network hiding inside the big one

A neural network is a large collection of numbers, called weights, wired so that data fed in one end produces an answer at the other. Training sets every weight to a small random value to begin with, its initialization. It then adjusts the weights over many passes through the examples until the network does its job, such as labeling a photograph of a handwritten 3 as the digit 3. A network for a task that simple can carry hundreds of thousands of weights.

Most of those weights turn out to be expendable. Pruning is the ordinary practice of deleting the weights that matter least once training is done, and networks tolerate a great deal of it. Remove the weights with the smallest magnitude, the ones closest to zero, and a network left with a small fraction of its original weights still labels its images about as accurately as the full one. That much was routine before 2019.

Jonathan Frankle and Michael Carbin, at MIT's CSAIL, asked a sharper question about that trimmed network. Their paper, "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks," won one of ICLR's two best-paper awards that year.¹² The pruned network is small, and small networks can be trained on their own. So could you skip the full network and train the small one from scratch? Try it the obvious way and it fails: the small network, started from fresh random weights, learns slower and ends up less accurate.¹

The paper found the one way to make it work. Start the small network from the exact weights it held at initialization, before any training, and it matches the full network. Frankle and Carbin call such a subnetwork a winning ticket, and the name spread across the field as shorthand for the idea that a trainable small network is hiding inside every big one.³

Prune, then reset to the original weights

The method has a name, iterative magnitude pruning, and it runs the train-and-prune loop several times over rather than pruning once.¹

1. Train

   Train the full network to completion, starting from its random initialization.

2. Prune

   Delete a fixed share of the surviving weights with the smallest magnitude, the ones nearest zero.

3. Reset

   Set every weight that survived back to the value it had at initialization, not the value training left it at.

4. Repeat

   Train the pruned network, prune again, reset again, until only a small fraction of the weights remain.

Step three is where the paper leaves ordinary pruning behind. The natural expectation is that pruning finds a good shape for the small network, a pattern of which weights to keep, and that you could then fill that shape with any reasonable starting weights and train it. Frankle and Carbin tested exactly that. They took a winning ticket, kept its shape, and reset its surviving weights to fresh random values instead of the originals. The result was worse: "when randomly reinitialized, winning tickets perform far worse," learning slower and topping out lower the more they were pruned.¹ The shape alone is not enough. The specific starting values the surviving weights were born with are part of what makes the small network trainable.

Small networks, and two ways to count sparsity

The result was demonstrated on small networks doing small vision tasks.¹ One of them was LeNet, a network of about 266,000 weights, learning to read the handwritten digits of MNIST. On the harder CIFAR-10 task, sorting small color photos into ten classes, they used a family of small convolutional networks they called Conv-2, Conv-4, and Conv-6. None of these are the billion-weight models that arguments about shrinking AI usually have in mind.

How far could they prune and still match the full network? Frankle and Carbin report sparsity as a quantity they call P_m, the percentage of weights that remain. Most later work reports it the other way, as the percentage of weights pruned away. The same network is "10 percent of weights remaining" in the first frame and "90 percent sparsity" in the second. Read a figure in the wrong frame and you have it exactly upside down.

Across the networks they tested, Frankle and Carbin found winning tickets smaller than 10 to 20 percent of the original size, which is 80 to 90 percent of the weights pruned away.¹ For LeNet on MNIST the ticket kept matching the full network's accuracy down to 3.6 percent of weights remaining, or 96.4 percent pruned. It learned fastest well before that extreme, around 13 to 21 percent remaining. The paper reads these percentages off its own figures rather than stating them to the decimal in the text, so the second decimal is its reported reading, not an independent recomputation.

The ticket is found after training, not before

To find a winning ticket you first train the full network, then prune it, then reset the survivors. The ticket is identified in retrospect, after the very training you were hoping to skip. Nothing in the paper tells you which small network will win before you have trained the big one.¹

This is the gap between what the paper proved and what "winning ticket" is taken to mean. The phrase suggests you could reach into a fresh, untrained network and pull out the lucky subnetwork cheaply, up front. The paper shows no such thing. It shows that the subnetwork exists and can be recovered by training and pruning, which costs more than training the network once, not less. Every matching subnetwork across this line of work, including the ones found later, is found the same way: by training and pruning.

The existence result held; the special-weights reading did not

The narrow claim, that a matching subnetwork exists and can be found by pruning, has held up and grown. The heavier readings stacked on top of it fared less well.

The 2019 paper flagged the scaling limit itself. On its two larger networks, VGG-19 and Resnet-18 on CIFAR-10, the plain procedure found no winning tickets at the standard learning rate: the pruned networks did no better than randomly reinitialized ones.¹ Frankle and Carbin recovered the result only by easing the network into training with a slow warmup. That seam is where the follow-up work went. In 2020 Frankle, with new coauthors, replaced the reset step.⁴ Instead of resetting the survivors all the way to their initialization, they rewound them only partway, to the values they had held a few percent into training. With that change, matching subnetworks appeared in Resnet-50 on ImageNet, a genuinely large network, at around 30 percent of weights pruned. Resetting to the very start had failed there.⁴ The clean statement "trainable from its original random initialization" is a small-network result. At scale, the object is trainable from an early training checkpoint, not from the start.

Two other papers went at the special-weights reading directly. Liu and colleagues retrained pruned networks from fresh random weights and found they did as well as networks that kept the large model's "important" weights.⁵ Tested against the lottery-ticket claim head-on, the winning-ticket initialization gave no advantage over random initialization at the standard learning rate. It helped only when the learning rate was turned down to 0.01, and that lower rate produced worse final accuracy than the standard 0.1.⁵ So the exact starting weights are not the free lunch the shorthand implies.

Zhou and colleagues asked how much of those starting weights actually matters, and answered: mostly just the sign.⁶ Keep whether each surviving weight started positive or negative, flatten all the magnitudes to a single constant, and the subnetwork still trains. Stranger still, the pruning mask carries real information on its own. Apply the mask to a fresh untrained network, adjust not a single weight, and it already classifies far above chance. For a ten-class task where guessing scores about 10 percent, that untrained masked network reached the accuracies below with no weight training at all.⁶

The phenomenon has also carried to the models people run now. In 2020 a team including Frankle found matching subnetworks in BERT, a widely used language model, at 40 to 90 percent of weights pruned across a range of tasks.⁷ Those BERT subnetworks are still found by iterative magnitude pruning, by training and pruning on the task, not read off cheaply before training.