The last post ended with token IDs. Now: how those integers become vectors, and why the embedding table is one of the most underrated parts of a model.
The lookup
An embedding table is a matrix of shape (vocab_size, hidden_dim). For Llama 3 8B, that's (128256, 4096), about 2GB in FP16. To embed a token, you just look up its row. Token 42 becomes row 42.
That's it. No computation, no neural magic. A table lookup. The "learning" happened during training, when the model figured out which tokens should be near each other in vector space.
Why vectors
The transformer operates on vectors, not integers. Every layer is matrix math on the hidden state, and the hidden state is a sequence of vectors. Embeddings are the bridge between the discrete world of tokens and the continuous world of linear algebra.
The property that matters: similar tokens have similar vectors. "cat" and "kitten" land near each other. That's what lets the attention mechanism find relationships, and what makes the whole thing more than a lookup table.
The embedding table is a dictionary where the definitions are vectors. "king" minus "man" plus "woman" equals "queen", and that arithmetic only works because the definitions live in a space where meaning is direction.
The inference angle
For inference, the embedding lookup is one of the cheapest operations in the whole forward pass. It's a single gather, not a matmul. But it's also one of the first memory touches, so it matters for cold-cache latency.
And it's a place where quantization shows up: the embedding table is often kept at higher precision than the rest of the model, because it's so sensitive. One more reason the table deserves respect.
The takeaway
An embedding is a token that learned where it lives. The table is the map, and the map is smaller than you think.
Next: attention, the thing everything hangs on.