The roofline was the map. This is the territory: CUDA kernels, the actual programs that run on the GPU. And the whole art of kernel engineering is three moves: launch, select, fuse.
Launch: the cost you forget
Every kernel launch has overhead: a few microseconds of CPU work to set up the grid, the blocks, the arguments. That's nothing on its own, but a transformer has hundreds of kernels per forward pass. Multiply and it adds up.
This is why kernel fusion matters so much. Instead of launching a separate kernel for the matmul, the bias add, the activation, and the next matmul, you fuse them into one kernel that does all of it in one launch, reading the intermediate values from fast on-chip memory instead of slow global memory.
Select: the right kernel for the shape
GPUs are not one-size-fits-all. A matmul with a 4096-wide inner dimension wants a different kernel than a matmul with a 64-wide one. A tiny batch wants a different kernel than a huge one. The art is selecting the right kernel for the shape, and the best libraries do this automatically.
This is why cuBLAS, CUTLASS, and Triton exist: they encode the selection logic so you don't have to. But understanding the selection is what lets you beat them when your shape is special.
Fuse: the win that compounds
Fusion is the big lever. FlashAttention is fusion at its finest: it fuses the entire attention computation into one kernel, avoiding the N×N materialization. The result: 3x speedup, not from better math, but from not moving data.
The principle: every time you avoid a trip to global memory, you win. Fusion is how you avoid the trips.
Kernels are like cooking. Launch is preheating the oven (overhead). Select is choosing the right pan for the dish. Fuse is making the whole meal in one pot so you don't wash dishes between courses.
Where this shows up
- Triton: write one kernel, get it compiled for any GPU. The selection is automatic.
- TensorRT-LLM: compiles the whole transformer into fused kernels. Maximum fusion, minimum flexibility.
- FlashAttention: the canonical fusion story, from the paper.
- CUTLASS: NVIDIA's library of hand-tuned kernels, the reference for what's possible.
The takeaway
Launch, select, fuse. Three moves, one goal: keep the GPU busy and the data local. Everything else is detail.
Next: model formats, the containers that carry the weights.