A token id is an address into a table the model learned

What the tokenizer hands the model

The previous step left the model holding integers. A tokenizer cut the text, matched each piece against its merge table, and handed back a sequence of token ids, nothing more. An id carries no numbers of its own; it is a position in a list. The step that turns that position into something a transformer can compute with is a single table lookup.

That table is a matrix. Call it E. It holds one row for every token in the vocabulary and one column for every dimension the model works in, so its shape is vocabulary size by model width. A transformer uses learned embeddings to convert each token into a vector of that width.¹ Voita states the mechanism plainly: for each vocabulary word a look-up table holds its embedding, and it is found using the word's index in the vocabulary.² Token id 2 means row 2 of E.

Reading row 2 by hand

To make the lookup concrete, shrink everything. Take a vocabulary of six tokens and a model width of four. The numbers below are chosen by hand to be small enough to check, not values any model produced. A real embedding row is learned; this one is invented for the arithmetic.

Token id 2 is the token [new]. Its embedding is row 2, read straight across: 0.6, 0.1, −0.5, 0.3, one number per column. Token id 0, the token [low], is row 0: 0.2, −0.4, 0.1, 0.5. There is no computation. The id is an address and the row is what sits at it.²

There is a longer way to reach the same four numbers, and it is what the linear-algebra picture means. Write id 2 as a one-hot vector, a length-six row that is 0 everywhere except a single 1 in position 2. Multiply it by E and each row is weighted by the matching entry of that vector, so every row but row 2 is multiplied by 0 and drops out, leaving row 2 exactly.² The matrix multiply and the lookup return the identical vector. No implementation performs the multiply. It reads the row.

One transformer-specific detail rides on top. Before the row enters the rest of the stack, every weight in it is multiplied by the square root of the model width.¹ On the toy that factor is the square root of four, which is two, so [new] enters as 1.2, 0.2, −1.0, 0.6. In the base transformer the width is 512 and the factor is about 22.6, so a real row is 128 times longer than this one. The scaling changes the size of the numbers, never which row was chosen.

Nobody arranged the rows

Those six rows were invented for the example. In a real model they begin as random numbers and are changed by training. The embedding matrix is parameters, adjusted by the same gradient signal that tunes every other weight in the network, and the rows that improve its predictions get reinforced. Olah, describing exactly this learned table, is careful about what the process produces.³

After enough training, a pattern appears in where the rows land. Tokens that show up in similar contexts end up with similar rows. This was never requested. In Olah's account the useful structure is a side effect: the training did not try to place similar words close together, and the arrangement more or less fell out of the optimization process.³ The rows in the toy table learned nothing, so any resemblance between two of them is coincidence. The claim about geometry rests on the trained models the sources measured, not on hand-picked numbers.

The most repeated illustration of that structure is an offset: the direction from one word's row to another can line up with a relationship. Mikolov's word2vec reported that the vector for king, minus the vector for man, plus the vector for woman, results in a vector closest to the one for queen.⁴ The result is closest to queen, not equal to it. Alammar, working the same example, records that king minus man plus woman does not exactly equal queen; queen is merely the nearest of the 400,000 vectors in his collection.⁵ The offset is a tendency the vectors exhibit, not an identity they satisfy.

The structure is real enough that it can be built on purpose, and seeing who builds it settles how much to trust it. Mikolov's later work found the space carried a linear structure, so that adding two vectors combined their meanings: the vector for Russia plus the vector for river lands near the vector for Volga River.⁶ GloVe, an independent construction from a different group, made the geometry an explicit goal, designing its vectors so that their differences capture the meaning set by a pair of words.⁷ In word2vec the geometry falls out of training on its own; in GloVe it is an objective the model is built to satisfy; and both are dedicated word-vector models, cleaner than a large model's input matrix, whose rows also feed everything above them. Gao and colleagues showed that a big model's learned embeddings tend to degenerate into a narrow cone, with low-frequency tokens crowded into one region.⁸ The tidy map of well-separated meanings is truest in the models made to produce it.

What the row is not

Holding the embedding apart from the things next to it matters as much as the lookup itself.

It is not the tokenizer. The tokenizer maps text to ids by counting merges in a corpus, a procedure fixed once and then frozen. The embedding maps ids to vectors by gradient descent, and its numbers keep moving throughout training.² One turns letters into integers by consulting a merge table; the other turns integers into vectors by consulting a learned one. Believing the embedding is part of the tokenizer fuses two tables built by different means, at different times, for different jobs.

It is not the output projection. The matrix that turns an id into a vector at the bottom of the model and the matrix that scores every vocabulary word at the top are different objects. Press and Wolf name them the input embedding and the output embedding, and their proposal is to tie the two, setting them equal so the model stores one matrix instead of two.⁹ Tying is a choice with a measured payoff: on their large model it cut parameters from 66 million to 51 million and lowered test perplexity from 78.4 to 74.3.⁹ The transformer adopts the same tie, sharing one weight matrix between the two embedding layers and the pre-softmax projection.¹ Shared weights do not collapse them into one layer. They sit at opposite ends of the network and do opposite work.

It does not carry position. The row for [new] is the same whether [new] is the first token in the line or the last. Word order reaches the model through a separate signal added on top. Vaswani's transformer gives its positional encoding the same width as the embedding precisely so the two can be summed, and adds it at the bottom of the stack.¹ A token's identity and its place in the sequence are two different vectors that happen to be added together, and separating them is the next piece of the machinery.

The rows training barely touched

The failure mode deserves the same care as the success, and it comes from the same mechanism. Gradient reaches a row only when its token appears in training. A token that sits in the vocabulary but almost never occurs in the training text keeps a row close to its random start, because little signal ever arrived to move it. Land and Bartolo, cataloguing these systematically, define them as tokens present in the tokenizer's vocabulary but nearly or entirely absent during model training, whose under-optimized embeddings can trigger anomalous or unintended behavior.¹⁰

The phenomenon was first noticed by accident. Rumbelow and Watkins, running k-means clustering on the embedding space of a GPT model, hit a set of strings sitting near the centroid of all 50,257 tokens that made no semantic sense as a group, among them the now-notorious SolidGoldMagikarp.¹¹ Asked to repeat these tokens, the model evaded, substituted other words, or produced different output on repeated tries at its most deterministic setting. Their explanation matches the mechanism: the strings had almost no involvement in training, so the model has no settled row to work from and does not know what to do.¹¹ This is the same rarity that bends the geometry. Gao's narrow cone is driven by low-frequency tokens crowding together, so the tokens the model saw least are both the strangest to prompt and the most distorted in the space.⁸

A row of numbers with a place

Follow the whole path once more. A tokenizer hands over an integer. The integer is an address into a matrix, and the row at that address is the token's embedding, four numbers in the toy and 512 in the base transformer. Nobody wrote those numbers down. They started random and were moved by training until, across the whole vocabulary, tokens used alike came to sit near one another, an arrangement no one designed, cleanest in the models built to produce it.³ The embedding is where a discrete symbol becomes a point in a continuous space, and the token's meaning is nothing more than where that point sits relative to the others. What it is not is the tokenizer that produced the id, the output layer that will later score the vocabulary, or the position signal added a step afterward. It is one learned row, and reading it is a memory access. Everything the transformer does next runs on those rows.