Skand Tandon
Standalone · Kernels

When the fusion you want is impossible, fuse into the producer

The activation I wanted to fold into a convolution's mainloop could not go there, for a reason specific to how Blackwell feeds its tensor cores. The answer was not to abandon the fusion. It was to apply it one kernel earlier, in the epilogue of whatever produced the input.

cutlass fusion blackwell

A pointwise activation sitting between two convolutions is the most obvious fusion target there is. It reads a tensor out of HBM, does a few flops per element, and writes it back. On paper you fold it into the consumer's prologue: the conv is going to load that data anyway, so apply the activation as it arrives and never materialise the intermediate at all.

On an SM100 implicit-GEMM convolution you cannot.

TMA_LOAD_IM2COL feeds UMMA directly. The tensor memory accelerator moves the im2col-transformed tile into shared memory and the tensor core consumes it from there. At no point does the data pass through registers under your control, so there is no hook to hang a prologue transform on.

This is not a missing feature. It is the mechanism that makes the kernel fast. The point of that load path is precisely that the data does not detour through the register file, so asking for a prologue hook is asking for the pipeline to be slower.

The second reason, which matters more

Suppose the hook existed anyway. The convolution had k=7, so each input element is loaded into seven different positions across the filter window. A prologue transform would recompute the activation seven times per element.

So the fusion I wanted was both impossible and, had it been possible, wasteful. That combination is worth noticing, because it says the transform is in the wrong place rather than that it cannot be fused. A value read k times and transformed k times should have been transformed once, before it was ever stored.

Fuse into the producer instead

The kernel that wrote the tensor wrote each element exactly once. Its epilogue is already a place where every output element passes through registers on the way to memory. That is the hook, and it is the correct one, because it runs at write cardinality rather than read cardinality.

Moving the activation into the upstream producer's epilogue did not shrink the standalone kernel. It removed it.

before   standalone activation kernel   8.8% of busy
after    zero launches, cost absorbed into
         a producer epilogue that was already
         writing every one of those elements

The same shape turned up a second time. Layout conversions were wrapped around every convolution, NCHW in and NHWC out and back again, because the filters were stored in the wrong layout. The fix was not to fuse the conversions or make them cheaper. It was to store the filters channels-last at load time, once, and delete all three legs. Bit-identical output, about 3.5% at high concurrency and 8.5% at low.

The general form

Both of those are the same move. Something is being done repeatedly at consumption time that could be done once at production time, or once at load time. The instinct is to make the repeated work cheaper. The better question is whether it belongs at that point in the pipeline at all.

When a fusion is impossible for a good reason, the good reason usually tells you where the fusion should have gone.

Sources