There's a world of difference between a kernel that works and a kernel that's optimized. NVIDIA's open-source kernel tooling is the map to the latter: a tile IR for tensor-core programming, a benchmarking library that measures the difference, and the frontends that make it usable.
cuda-tile: the IR for tensor cores
cuda-tile is an MLIR-based intermediate representation and compiler infrastructure for CUDA kernel optimization. It focuses on tile-based computation patterns and optimizations targeting NVIDIA tensor core units. The idea: express your kernel as tiles, let the compiler lower it to the right MMA instructions for the target architecture.
Why it matters: hand-writing tensor-core kernels is error-prone and architecture-specific. A tile IR lets you express the computation once and target H100, B200, and whatever comes next.
nvbench: measure before you optimize
nvbench is NVIDIA's CUDA kernel benchmarking library. It's the discipline of kernel engineering made tool: write a benchmark, run it across architectures, and get the numbers that tell you whether your optimization actually helped.
Why it matters: the fastest way to a slow kernel is to guess. The fastest way to a fast kernel is to measure, change one thing, measure again.
cudnn-frontend: the kernel that powers everything
cudnn-frontend is NVIDIA's modern, open-source entry point to cuDNN, the deep learning primitive library. It exposes the same kernels that power cuDNN's fastest paths, with a programmable API.
A note on NV-Kernels
If you search for "NV-Kernels" you'll find NVIDIA's Ubuntu kernel repo, which ships kernels optimized for NVIDIA server systems. That's an OS-level kernel, not a CUDA kernel library. The distinction matters: one is the Linux kernel tuned for DGX-class hardware, the other is the GPU code that runs your model. Both are worth knowing.
The kernel craft
Reading this ecosystem is a masterclass in the craft of kernel optimization:
- Memory coalescing - arranging access so each warp reads contiguous memory, not scattered words.
- Occupancy vs ILP - balancing how many warps are resident against how much work each warp does.
- Tensor core shapes - picking the right MMA instruction shape for the data.
- Shared memory tiling - reusing data from shared memory instead of hammering global.
These are the same moves I wrote about in "the craft of kernels", except here they're production-grade and battle-tested.
In production
For a serving engine, this tooling is the difference between "works" and "wins benchmarks". The tile IR lets you target new architectures without rewriting kernels. nvbench gives you the numbers to justify every optimization. And because they're open, you can verify what they do instead of trusting a black-box library.
The takeaway
Open-source kernel tooling is the new standard. When you can express a kernel as tiles, measure it with nvbench, and target any architecture, there's no excuse for treating optimization as a black box.
Sources
- cuda-tile repo: the MLIR-based tile IR for tensor-core kernels.
- nvbench repo: the CUDA kernel benchmarking library.
- cudnn-frontend repo: the open-source entry point to cuDNN.
- NV-Kernels repo: Ubuntu kernels optimized for NVIDIA server systems.
- TensorRT-LLM repo: where these kernels ship in a serving runtime.
- Model-Optimizer repo: the tooling that prepares models for these kernels.