transformers-first-principles
collection · 2 published
transformers-first-principles
A transformer never reads the letters in strawberry
The model reads integer tokens from a table built by counting letter pairs across a corpus, and that table decides how it handles spelling, digits, and rare scripts.
transformers-first-principles
A token id is an address into a table the model learned
The integer a tokenizer hands over indexes one row of a learned matrix, and where that row sits in space is the only place its meaning lives.