king - man + woman = queen scored 65.6% on word2vec's own test

The sentence everyone quotes

The one line people quote from word2vec is an equation. Take the vector for king, subtract the vector for man, add the vector for woman, and the result lands nearest the vector for queen.¹ Tomas Mikolov and four colleagues at Google published it in 2013, first in a paper called "Efficient Estimation of Word Representations in Vector Space"¹ and then in a companion paper² at the NeurIPS conference that December.³

The line got repeated as proof that the model had learned what the words mean. It is a real result. It is also a number: on the analogy test the authors built for the purpose, the arithmetic recovered the intended fourth word 65.6% of the time.¹

Two models that predict their neighbors

word2vec is two small models, CBOW and Skip-gram. Both learn a single vector for each word from raw running text, the kind of word vector this course covers on its own. What is worth pinning down here is how they learn it: by prediction, and nothing else. CBOW takes the words around a gap and predicts the word in the gap, averaging the surrounding context and ignoring its order.¹ Skip-gram runs the same idea backward, predicting the likely neighbors from the one word in the middle.¹ Neither model is told what any word means. Each just nudges a word's vector until words that keep the same company end up close together.

What stunned people in 2013 was the price. Earlier neural language models pushed the text through a hidden layer that cost most of the compute. word2vec throws that layer out, which is what lets it train on billions of words on a single machine in under a day.¹ The companion paper trims the cost again with two shortcuts. Negative sampling scores each training example against a handful of randomly drawn words instead of the entire vocabulary. Subsampling randomly drops the most frequent words like "the" before training.² Turning that much text into usable vectors on ordinary hardware is why everyone could suddenly build on the result, and that part has never been in doubt.

One test, 19,544 questions

The famous claim rests on a test the authors wrote themselves. It holds 19,544 questions, 8,869 about meaning and 10,675 about grammar, sorted into 14 categories.¹ Each question gives three words and asks for the fourth: Athens is to Greece as Oslo is to what, or "walking" is to "walked" as "swimming" is to what. The model answers with the same vector arithmetic, and the score is the fraction it gets exactly right.

The best score in the first paper was 65.6%, split almost evenly into 66.1% on the meaning questions and 65.1% on the grammar ones. That run used Skip-gram with 1,000-dimensional vectors, trained on a 6-billion-word Google News corpus with the vocabulary capped at the million most frequent words.¹ The companion paper's best word-analogy score was 61%, from Skip-gram with 300-dimensional vectors on a roughly one-billion-word news set.² The score depends heavily on which training shortcut you pick.

One number from this paper gets misquoted constantly. The companion also reports 72%, and that figure travels as if it were the analogy accuracy. It belongs to a different test: a set of phrase analogies, trained on a much larger corpus of about 33 billion words, not the 19,544-question word set.² The word-analogy result is 65.6% in the first paper and 61% in the second. Keep the 72% with the phrases, where it was earned.

What the arithmetic is doing

So the trick works about two times in three. What is it doing on the times it works? Two later papers took the operation apart, and both are worth reading in full.

Omer Levy and Yoav Goldberg did the algebra in 2014.⁴ The result is quieter than the story around it.

Levy and Goldberg then showed the same regularities in plain count-based vectors, the kind built by tallying how often words occur near each other with no neural network at all. Those sparse vectors scored 68% with a multiplicative variant, beating the neural embedding's 66.7% on the same task.⁴ The pattern lives in the co-occurrence counts. The network preserves it rather than inventing it.

Tal Linzen pushed on the skip-the-inputs step in 2016.⁵ Run the literal arithmetic without hiding the inputs, and the nearest word to king minus man plus woman is king itself 93% of the time, and one of the other two inputs another 5%. That is an input word 98% of the time.⁵ Excluding those words is a hand-built rule, not something the geometry does on its own. And once you do exclude them, a baseline that ignores the offset entirely and simply returns the nearest neighbor of the third word already captures much of the score, reaching 70% on the plural questions.⁵ Reverse the direction of an analogy and accuracy falls on average, and it falls in step with that offset-free baseline rather than with the offset.⁵

None of this makes the 65.6% fake. The vectors really do place related words in consistent directions, so that countries sit near their capitals in a roughly repeatable way.² The regularity is real. The reasoning is the part we added.

What held up

The durable part was the embedding, not the analogy trick. A dense vector learned from raw text became the standard first layer of language systems, and the prediction recipe was matched and passed almost at once. Within a year, GloVe, from a Stanford group, reached comparable scores on the very same analogy set by factoring a global word-co-occurrence table instead of predicting from local windows.⁶ That a rival method aimed at the same 19,544 questions tells you the field had adopted the Google set as its shared yardstick and had already moved past word2vec's exact approach.

What survives from 2013 is the finding that you can learn a useful vector for each word by predicting its neighbors, cheaply and at scale. What did not survive a close read is the shorthand that the arithmetic on those vectors thinks.