LLM inference mechanics

The first post was philosophy. This one is plumbing. Before you can optimize inference, you need to know what a model actually does with your prompt, step by step, and where the seconds go.

Here's the whole journey of one token, from your keyboard to the response.

The pipeline

1. Tokenize - your text becomes integers. "Hello world" becomes a few IDs from a vocabulary of 50,000 to 200,000 tokens. This is fast, microseconds.
2. Embed - each token ID becomes a vector. A lookup table maps ID to a dense vector of dimension d (say 4096). This is also fast.
3. Transformer blocks - the slow part. Each block does attention + feed-forward. Attention lets tokens talk to each other; the feed-forward lets each token think alone.
4. Unembed - the final hidden state becomes a probability distribution over the vocabulary.
5. Sample - pick the next token (greedy, top-k, temperature, whatever).

Then the new token joins the sequence and you repeat from step 3. That loop is the entire job. Everything in inference engineering is about making this loop faster, cheaper, or more reliable.

Where the time goes

Not all steps are equal. Tokenization and embedding are negligible. The transformer blocks are essentially all of it. And within a block, the split depends on the phase:

• Prefill (first token): the prompt is processed in parallel, so the transformer blocks do huge matrix multiplications. Compute-bound.
• Decode (subsequent tokens): one token at a time, and each step re-reads the weights. Memory-bound.

This is the same asymmetry I'll come back to. For now, the key insight is that decode is the loop that runs once per token, and it's the loop that determines how fast the model feels.

A worked example

Take a 7B parameter model in FP16. Weights are 14 GB. On an H100 with 3.35 TB/s bandwidth, just reading the weights once takes:

14 GB / 3.35 TB/s ≈ 4.2 ms

That's the floor for one decode step: about 4 milliseconds per token, even if the compute were free. In practice it's a bit more, but this is why a 7B model on an H100 does roughly 200-250 tokens/sec in the best case. The math is inescapable.

Why this matters for the journey

Every optimization we'll meet over the next 99 days is a variation on one theme: read the weights less, or read them in a cheaper format.

• Quantization makes the weights smaller (fewer bytes to read).
• Batching amortizes one weight read across many tokens.
• KV-cache management avoids re-reading what we already know.
• Speculative decoding spends idle compute to avoid extra weight reads.

Keep that lens on and the whole field becomes legible.

Tomorrow: what's actually inside the model, and why "weights" is doing so much heavy lifting.