Ops:byte ratio and the roofline

I spent my first months in inference thinking the job was "make the model faster." It took me embarrassingly long to realize the real job is figuring out what's actually slowing the model down, because if you optimize the wrong resource, you can burn a week and change nothing.

This is the framework I wish I'd had on day one. Two numbers, one chart, and a rule of thumb that explains more about LLM inference than any single paragraph I've read.

Your GPU has two resources, and only two

Strip away the marketing and an accelerator is just a machine with two relevant properties:

• Compute - how many floating-point operations per second (FLOP/s).
• Memory bandwidth - how many bytes per second (B/s).

Everything else, Tensor Cores, HBM, NVLink, L2 cache, is detail layered on top of those two numbers. And here's the thing: they are never balanced. Every GPU has a fixed ratio between them, called the ops:byte ratio.

Take an H100 in FP16: roughly 989 teraFLOPS of dense compute against 3.35 TB/s of bandwidth. That's an ops:byte ratio of about 295. The GPU can do 295 operations for every byte it reads. For inference to run at full speed on an H100, your algorithm needs to do 295 FLOP per byte accessed. Less than that, and memory is the bottleneck. More, and compute is.

Arithmetic intensity: the algorithm's side of the deal

Ops:byte ratio is a hardware property. Its software mirror is arithmetic intensity, the number of FLOP an algorithm performs per byte of memory it touches. Compute it as work divided by memory traffic for whatever operation you're analyzing.

Plot performance against arithmetic intensity and you get the roofline model: a diagonal line (the memory bandwidth ceiling) meeting a horizontal line (the compute ceiling) at the ridge, the ops:byte ratio. Any algorithm lands on one side or the other:

• Compute-bound: above the ridge, pinned to the horizontal ceiling. More FLOPS = faster.
• Memory-bound: below the ridge, pinned to the diagonal. More bandwidth = faster.

This is the single most useful chart in inference. Memorize its shape.

Why LLMs are slow: the two phases

Every LLM request has two phases, and they land on opposite sides of the roofline:

• Prefill - the model reads the whole prompt in parallel and produces the first token. Big matrix multiplications, lots of compute per byte. Compute-bound. Sets your time-to-first-token (TTFT).
• Decode - the model produces tokens one at a time. Each step loads the entire model weights from memory to produce one token. Memory-bound. Sets your tokens-per-second (TPS).

In prefill, the weights are loaded once and then hammered with a series of large matmuls, high arithmetic intensity, so it rides the compute ceiling. In decode, the weights are loaded again for every single token, and each token is produced by a cheap vector-matrix multiply. You're moving gigabytes of weights to compute a few hundred FLOP. The conveyor belt is the bottleneck.

This asymmetry is why batching works: process a batch of requests and you reuse the same weight bytes across many tokens, raising arithmetic intensity and pulling decode up off the memory ceiling. It's also why speculative decoding works: the spare compute during memory-bound decode can be spent on cheap draft tokens. Everything good in inference traces back to this chart.

The math, worked by hand

To make this concrete, let's compute the arithmetic intensity of a single decode step for a model with attention head dimension d=128 at sequence length N=4096, using standard attention (no FlashAttention tricks), in FP16 (2 bytes per value).

Standard attention does three matrix operations:

# Q, K, V are N x d (4096 x 128)
S = Q @ K.T      # (N x N) scores
P = softmax(S)   # (N x N) probabilities
O = P @ V        # (N x d) output

Memory traffic (reads + writes), in elements:

• Read Q, K, V: 3 × N×d = 3 × 4096×128 = 1,572,864
• Write S: N×N = 16,777,216
• Read S, P for softmax: 2 × N×N = 33,554,432
• Write P: N×N = 16,777,216
• Read P, V for O: N×N + N×d = 16,777,216 + 524,288
• Write O: N×d = 524,288

Total memory movement ≈ 86 MB (43M elements × 2 bytes). Total compute ≈ 2×N×N×d for the two matmuls = 2 × 16.7M × 128 ≈ 4.3 GFLOP.

Arithmetic intensity = 4.3e9 / 86e6 ≈ 50 FLOP/byte.

An H100's ops:byte ratio is 295. Decode's arithmetic intensity is 50, six times below the ridge. It's firmly memory-bound. No amount of compute optimization will make decode faster, because decode isn't waiting on compute. It's waiting on the conveyor belt.

Play with it yourself

Drag the sliders. This is the same math, interactive:

The rule of thumb

If it's memory-bound, quantize or batch. If it's compute-bound, use lower precision or better kernels. If you don't know which, you're guessing, and guessing is how you burn a week.

Every optimization in this field is either "raise the arithmetic intensity" (batching, fusion, tiling), "lower the bytes" (quantization, KV-cache compression), or "move the ridge" (better hardware). Once you see the chart, you see the whole discipline.

Next up: the KV cache, the memory hog that makes decode memory-bound in the first place.