A token walked through the transformer. Now I'm zooming into the three things a model is actually made of: weights, tokens, and logits. Understand these three and you understand 90% of what's happening on the GPU.
Weights: the learned part
Every model is a stack of matrices. The weights are the numbers in those matrices, learned during training. For a 7B model in FP16, that's about 14GB of weights, which is why it needs a GPU with enough VRAM.
The key property: weights are static. They don't change during inference. That's what makes quantization possible, and it's why you can cache them, fuse them, and store them in the fastest memory.
Tokens: the input and output
Tokens are integers, each mapping to a word piece via the tokenizer (I build one from scratch later). The input is a sequence of token IDs. The output is a sequence of token IDs. Everything in between is math on those IDs.
For inference, the interesting number is the token count. It drives the KV cache size, the prefill time, and the cost. Tokens are the currency of inference.
Logits: the raw predictions
After the final layer, the model outputs a vector of logits, one per token in the vocabulary. A logit is an unnormalized score. The softmax turns them into probabilities, and the sampler picks the next token.
Here's the thing people miss: the logits vector is as big as the vocabulary. For a 128K vocab, that's 128K floats per token, which is why the final layer is a bottleneck and why sampling matters.
Weights are the sheet music, tokens are the notes being played, logits are the audience's applause before it's normalized. The transformer is the orchestra.
The flow
- Tokenize the prompt into token IDs.
- Embed the IDs into vectors.
- Run them through the transformer layers, attending.
- Project the final hidden state to logits.
- Softmax, sample, append the new token, repeat.
Every optimization in this journey is a variation on this flow. Quantization shrinks the weights. Speculative decoding makes step 5 cheaper. Prefix caching skips steps 1-3 for shared prefixes. The roofline tells you which step is the bottleneck.
The takeaway
Weights are the static truth, tokens are the moving parts, logits are the raw guess. Everything else is optimization on top.
Next: embeddings, where integers become vectors.