The theory was launch, select, fuse. This is the practice: what it actually feels like to take a kernel from "works" to "fast", and the loop that gets you there.
Step one: profile, don't guess
Every kernel optimization starts the same way, with a profiler. Nsight Systems shows you the timeline: where the GPU was idle, where the kernels were, where the gaps are. Nsight Compute shows you the kernel internals: occupancy, memory throughput, compute throughput, warp stalls.
The first lesson: your intuition about the bottleneck is usually wrong. You'll guess "it's the math" and the profiler will say "it's the memory". You'll guess "it's the kernel" and it'll say "it's the launch overhead". Profile first, always.
Step two: find the memory moves
Once you know it's memory-bound, the question is where the bytes go. Every tensor read from global memory is a trip. Every write is a trip. The kernel's job is to minimize trips.
The moves that matter, in order of payoff:
- Coalescing: make adjacent threads read adjacent memory. A coalesced read moves 128 bytes per transaction; an uncoalesced one moves 4. That's a 32x difference in effective bandwidth.
- Tiling: load a tile into shared memory once, reuse it many times. This is the FlashAttention move.
- Fusion: don't write an intermediate to global memory if the next kernel can consume it from registers or shared memory.
Coalescing alone is often a 2-5x win. Tiling is another 2-3x. Fusion is the compounding one.
Step three: the occupancy trade
Occupancy is how many warps are ready to run. High occupancy hides latency: when one warp waits on memory, another computes. But occupancy costs resources: more warps means less shared memory and fewer registers per warp.
The art is the trade. A kernel with 100% occupancy but no shared memory is slower than one with 50% occupancy and a good tile. The profiler tells you which you're in.
Writing a fast kernel is like running a kitchen. Occupancy is how many cooks you have. Shared memory is the counter space. The roofline is the menu. You can't fix a slow kitchen by adding cooks if the counter is the bottleneck.
The loop
One thing at a time. Re-profile after each. The temptation is to change three things and hope, but then you don't know which worked. The discipline of one change per iteration is what makes the loop converge.
The moment it clicks
There's a moment in every kernel project where the roofline stops being a theory. You profile, you see the kernel is at 17% of peak, you fix the coalescing, and suddenly it's at 60%. The math was always there; the memory was the wall. That moment is why I do this.
The takeaway
A fast kernel isn't written, it's profiled into existence. One bottleneck at a time, until the roofline is satisfied.