A transformer redraws a word's vector for every sentence it appears in

A word enters as a number before it means anything

Ask almost any chatbot for words like dog and it offers puppy before it offers cat. Feed the right tool the arithmetic "king minus man plus woman" and it hands back queen. Both are the same trick. Somewhere inside the model, dog and puppy sit close together. And king, man, woman, and queen sit at points that can be added and subtracted like ordinary numbers. A word, though, is just a string of letters. Getting from letters to something a model can do arithmetic on takes a specific, traceable series of steps.

The first step turns text into a short sequence of integers, one per token, by matching each piece against a fixed vocabulary list. That integer is where this lesson picks up. On its own, an integer carries no meaning: it is a position in a list, nothing else. The step that gives it one is a second lookup, into a much larger table. Vaswani and colleagues describe the transformer architecture almost every current large model is built on. They state the mechanism in one line: the model uses learned embeddings to convert each input token into a vector of a fixed width.¹ Token id 40, whatever token that happens to be, means row 40 of that table. Nothing is computed. The row is read.

That is as far down as this explanation goes. Every step before this one had a further mechanism underneath it, worth asking about. This one doesn't. The row is not standing in for some deeper process. Reading it out of the table is the entire operation. The vector it returns, a few hundred to a few thousand numbers, is what the rest of the model works with from here on.

The table nobody wrote

Picture the table itself: one row per token the model knows, one column per number in the vector, usually called the model's width. In the base transformer Vaswani's paper describes, the width is 512 numbers; in the larger version, 1024. The vocabulary runs to roughly 32,000 to 37,000 tokens, depending on the language pair.¹

None of those numbers describes anything a person decided in advance. Before 2013, the field's standard practice was to keep word vectors deliberately small. Mikolov and colleagues introduced word2vec, which popularized much larger ones. They describe the prior norm as a "modest dimensionality of the word vectors between 50-100."² A vector that size starts as random numbers. Training nudges it, one example at a time, by the same process that adjusts every other weight in the network. Nobody writes a row down. Training arrives at it.

What training arrives at is at least partly checkable. A real 50-number word vector, the kind word2vec's contemporaries were producing, begins 0.50451, 0.68607, -0.59517 and runs for 47 more entries. Jay Alammar, showing that vector for the word king, says plainly: "It's a list of numbers. We can't tell much by looking at the values."³ Printing every number in a row is one thing; explaining what any single one means is another. None of the papers behind this lesson claim the second, only where a row sits relative to the others.

The arithmetic word2vec actually showed

Once every word is a row, "arithmetic on words" is literal: take one row, subtract another, add a third, and search the table for the nearest row to what is left. Mikolov and colleagues built exactly that search. Their own worked table shows the offset from Paris to France, added to Italy, landing nearest Rome, and added to Japan, nearest Tokyo. The same offset run on professions turns Einstein minus scientist plus Messi into midfielder, and plus Picasso into painter.² Across a test set of 19,544 such questions, their best model, trained on six billion words with 1,000-number rows, solved 65.6 percent overall. That included 66.1 percent of the semantic questions and 65.1 percent of the syntactic ones.²

The sentence most people actually quote, king minus man plus woman equals queen, is not in that table, or anywhere else in this paper. It appears once, in a single sentence, credited to a separate paper by overlapping authors.² The companion paper published the same year, introducing the training tricks behind the numbers above, never uses the words king or queen. Its own headline example is that the vector for Madrid, minus Spain, plus France, lands closer to Paris than to any other word vector in the collection.⁴ The paper that actually puts the literal king-queen sentence into print is a competing method: Pennington, Socher, and Manning's GloVe, published the following year. It uses the sentence to motivate its own evaluation scheme, while crediting the same third, uninspected paper for the underlying idea.⁵ The example everyone reaches for to explain this trick is real; it just doesn't come from where the story says it does.

GloVe's method is built independently, from a global count of which words appear near which others rather than word2vec's local prediction. It reports a stronger result on the same kind of test: 75 percent overall, its best run using 300-number rows trained on 42 billion words. That is three questions in four.⁵

How much of that is really arithmetic

A 75 percent score sounds like the vector space is doing real geometric reasoning. Two independent researchers checked how much of that number came from arithmetic, and how much came from a rule built into the test. Tal Linzen audited the method from outside it. A literal, unassisted version, one that simply reports whichever word sits nearest the computed point, mostly hands back one of the three words already in the question. Subtract man from king, add woman, and the nearest row to that point is often king or woman itself, not queen.⁶ Every reported accuracy figure above comes from a version of the test that forbids the search from answering with a word already in the question. Reversing the direction of a question changes the score, even though the underlying arithmetic does not. Linzen found the same offset, asked backward, dropped a US-cities category's accuracy from 69 percent to 17 percent.⁶

Nissim, van Noord, and van der Goot measured exactly how much the forbidding rule was carrying. On the standard test set, the same constrained search behind every accuracy number above scores 0.74 by one measure and 0.71 by a stricter one. Strip the rule out, let the search answer with an input word if that is genuinely nearest, and the identical formula collapses to 0.21 by both measures.⁷ Three answers in four become roughly one in five. None of this means the vectors carry no structure. Something in that space does put queen nearer than most words to that point, or the constrained search could not find it at all. It means a meaningful share of the trick's fame belongs to the test's rulebook, not to the geometry alone.

A vector that moves with the sentence

Word2vec and GloVe share one more property, easy to miss because the demo depends on it. Each word gets exactly one row, decided once during training, and reused every time the word appears again. The row for play is the same row whether the sentence is about baseball or Broadway.

A transformer's embedding table starts the same way, a static row per token, read the same way Vaswani's paper describes.¹ But that row is not the end of its story. The model immediately hands it to attention, layer after layer. Each layer recombines a token's row with the rows around it in that particular sentence, a step this lesson names but leaves for its own lesson to explain. Peters and colleagues, building a language model, showed exactly what that recombination buys. Where word2vec-style approaches "only allow a single context-independent representation for each word," their model assigns each token a vector that is a function of the entire sentence around it.⁸ Put a fixed vector for play next to its nearest neighbors and the senses blur together: playing, game, games, played, football, multiplayer, among others.⁸ Ask the sentence-sensitive version the same question with two different sentences: "Chico Ruiz made a spectacular play on Alusik's grounder" and "Olivia De Havilland signed to do a Broadway play for Garson." It answers with two different vectors, each correctly matched to sentences about the same sense of the word.⁸

The real difference between the 2013 story and a modern model is not the arithmetic. It is what happens to the row immediately afterward. Word2vec's vectors stop at the table: reading the row is the last step as well as the first, so a demo built from those rows tests the whole representation a word will ever get. A transformer's row is only the first step. Pull it up for play in isolation, before attention has touched it, and it tells you as little as Alammar's fifty numbers did. Which sense is meant depends on the sentence, and the sentence has not been consulted yet.