Skand Tandon
Standalone · Kernels

Grid 24 is not headroom

Two kernels took a third of the busy time on under a fifth of the GPU. That reads as idle silicon waiting to be filled, and it is the most expensive misreading I have made on Blackwell. The grid was small because occupancy was already closed, and every hour spent trying to fill those SMs was spent against a wall.

cutlass blackwell occupancy

A profile of a decoder I was tuning produced a finding that was correct, specific, and pointed me straight at a wall. Two CUTLASS kernels dominated, and both ran on a small fraction of the part:

cutlass ConvUniversal   grid  24   23.6% of busy
cutlass GemmUniversal   grid  12   11.0%
hand-written activation grid 660    5.9%

148 SMs on the part, so about 35% of busy time
lands on under a fifth of the GPU

The reading almost writes itself. There are idle SMs, so run more work concurrently and fill them. I tried exactly that with a second inference context and got minus 0.4% for 39 GB of extra VRAM. Then I needed a second analysis just to explain why the first one had failed, which is usually the sign that the question was wrong rather than the answer.

The explanation was that the two contexts overlapped for only 13% of kernel time, and the device was already 93% busy through plain alternation. There was no idle device to recover. The SMs were idle. The GPU was not.

The question I never asked

I had treated grid 24 as a given, a property of the problem to work around. It is not a given. It is a consequence, and the consequence points somewhere completely different from where I was looking.

Grid size here is the tile count, and the tile is chosen by the kernel's resource footprint. So the question is not how do I get more CTAs resident. It is how many CTAs can be resident at all, and that is arithmetic you can do before running anything.

Shared memory first. 230,400 bytes per CTA against roughly 227 KB usable per SM gives exactly one CTA per SM. That alone caps it.

Then registers, which bind harder. 155 registers per thread across 256 threads is 40,960 registers per CTA. Two CTAs would need 81,920 of the SM's 65,536. Even if you freed every byte of shared memory, the register file says no.

That is the whole answer and it took minutes once I thought to ask. Occupancy on this kernel is not low. It is closed. No configuration raises it, because both resources bind, registers first.

I checked it three ways before believing it, which I would recommend when the conclusion tells you to stop doing the obvious thing: the occupancy limiter section in Nsight Compute, the register arithmetic above, and a two-stream grid sweep that found no concurrency benefit at any grid size.

The counters were saying it too

Once I knew what to look for, the profile had already answered the question somewhere else:

sm_throughput      31.6%
DRAM throughput     3.1% of peak
dominant stall     long_scoreboard

Low on compute and low on bandwidth, with the dominant stall being a wait on memory dependencies. That is not a compute-bound kernel and it is not a bandwidth-bound one. It is latency-bound, and latency-bound at one CTA per SM is a specific situation with a very short list of remedies.

What is left when occupancy is gone

You normally hide memory latency two ways: run other warps while one waits, or go deeper in the software pipeline so the data for iteration n+k is already in flight. At one CTA per SM the first one is unavailable. Mainloop stage depth is the only latency-hiding mechanism there is.

Which inverts the whole optimisation. Stages are bought with shared memory. So anything that frees shared memory in order to raise occupancy spends the one lever that works to buy a lever that does not exist. I swept five tile sizes and throughput was monotonic in shared memory per CTA: the largest tile that fits was fastest, and every smaller one was worse, in order.

I wrote do not spend further cycles on conv occupancy into my own notes, which was the most useful sentence I produced that week.

What I would do differently

A profiler tells you what a kernel costs. It will not tell you why the kernel is shaped that way, and on an already-tuned kernel that second question is the only one left.

Sources