TensorRT-LLM vs eager PyTorch

When I first started running LLMs in production, I used the simplest path available: load the Hugging Face checkpoint, call model.generate(), and let PyTorch do its thing. It works. It is also leaving a huge amount of performance on the table.

TensorRT-LLM takes a fundamentally different approach. Instead of interpreting a Python computation graph at runtime, it compiles the entire model into an optimized engine ahead of time. The result is a binary blob that the GPU executes directly, with fused kernels, quantized weights, and a memory layout tuned for the specific GPU you are targeting.

Today I want to walk through what actually changes when you move from eager PyTorch to TensorRT-LLM, and why the performance gap is as large as it is.

How eager PyTorch executes

In eager mode, PyTorch dispatches operations one at a time. Each nn.Linear, each activation function, each layer norm is a separate kernel launch. The execution flow looks like this:

1. Python calls into the C++ dispatcher.
2. The dispatcher selects a CUDA kernel.
3. The kernel runs on the GPU.
4. The result is written back to GPU memory.
5. Python regains control and dispatches the next operation.

The problem is step 4 and 5. Between every kernel, intermediate results are written to HBM and then read back for the next operation. For a transformer block, a single forward pass might launch dozens of kernels, each paying the cost of a kernel launch (a few microseconds) and a round trip to global memory.

# Eager PyTorch: each op is a separate kernel
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
model = model.half().cuda()
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")

inputs = tokenizer("The capital of France is", return_tensors="pt").to("cuda")

# This dispatches hundreds of individual CUDA kernels
with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=128)

This works, but it is fundamentally inefficient for inference. Every intermediate activation hits HBM, and kernel launch overhead accumulates across the 32 (or 80) transformer layers.

What TensorRT-LLM does differently

TensorRT-LLM takes the model through a multi-stage compilation pipeline:

• Graph construction: The model is defined using TensorRT-LLM's own layer API (not PyTorch's nn.Module). This produces a static computation graph.
• Graph optimization: TensorRT applies layer fusion, constant folding, and dead code elimination. Adjacent operations like linear + bias + activation get merged into a single kernel.
• Kernel selection: For each fused operation, TensorRT benchmarks multiple kernel implementations and picks the fastest for your specific GPU. This is why builds take a while: it is literally running timing experiments.
• Memory planning: The engine pre-allocates all activation memory and reuses buffers across layers, eliminating dynamic allocation overhead.
• Serialization: The result is a binary engine file that can be loaded and executed without any Python overhead.

# Build a TensorRT-LLM engine (simplified)
# This happens offline, once per model + GPU combination

trtllm-build \
  --checkpoint_dir ./llama-7b-ckpt \
  --output_dir ./llama-7b-engine \
  --gemm_plugin float16 \
  --gpt_attention_plugin float16 \
  --max_batch_size 8 \
  --max_input_len 2048 \
  --max_seq_len 4096

Where the speedup comes from

The performance gains are not magical. They come from a few specific, well-understood optimizations:

Kernel fusion. In eager mode, a typical attention block runs separate kernels for QKV projection, rotary position encoding, scaled dot-product attention, and the output projection. TensorRT-LLM fuses these into far fewer kernel launches. The GPT attention plugin, for example, combines the entire multi-head attention computation (including KV cache management) into a single kernel. Fewer launches means less overhead, and keeping intermediate values in registers or shared memory instead of HBM means less memory traffic.

Custom GEMM kernels. Matrix multiplications dominate transformer compute. TensorRT-LLM ships with hand-tuned GEMM kernels that exploit the specific Tensor Core layout of each GPU generation. On Hopper GPUs, these kernels use the TMA (Tensor Memory Accelerator) for asynchronous data movement and wgmma instructions for warp-group matrix multiply. The auto-tuning step during build selects the best tile sizes and pipeline depths for your exact shapes.

In-flight batching. TensorRT-LLM's runtime natively supports continuous batching. New requests can join a running batch between decode steps, and finished requests release their slots immediately. In eager PyTorch, implementing this requires significant custom code around the generation loop.

Quantization as a first-class citizen. When you build with --use_fp8 or --use_weight_only_int8, quantization is baked into the fused kernels. The weights are stored in the quantized format, and dequantization happens inside the GEMM kernel itself, not as a separate pass. This is substantially faster than the "dequantize then matmul" pattern you see in many PyTorch quantization approaches.

The cost: build time and flexibility

Nothing is free. TensorRT-LLM has real costs:

• Build time. Compiling a 7B model engine takes 5 to 15 minutes. A 70B model with tensor parallelism can take 30 minutes or more. You need access to the target GPU during build, because the auto-tuning profiles kernels on real hardware.
• Static shapes. The engine is built for specific maximum sequence lengths and batch sizes. You declare max_input_len, max_seq_len, and max_batch_size at build time. Exceeding these at runtime is not possible; you need to rebuild.
• Debugging difficulty. When something goes wrong with an eager PyTorch model, you can set breakpoints and inspect tensors. With a compiled engine, you are debugging a binary blob. Numerical mismatches between the engine and the reference implementation require careful layer-by-layer comparison.
• Model support lag. New model architectures appear on Hugging Face daily. TensorRT-LLM requires explicit support for each architecture in its model definition layer. There is always a gap between when a new model is released and when TensorRT-LLM supports it.

When to use which

My rule of thumb has become straightforward:

Use eager PyTorch for research, prototyping, and any situation where you need to modify the model frequently. Use TensorRT-LLM when you have a fixed model, a known GPU target, and throughput matters.

In practice, most production deployments I have worked on follow a pattern: develop and validate with PyTorch, then compile to TensorRT-LLM for the serving deployment. The compilation step fits naturally into a CI/CD pipeline, where the engine is built as an artifact alongside the container image.

It is worth noting that vLLM and SGLang have closed some of this gap by using torch.compile and custom CUDA kernels (like FlashAttention and FlashInfer). They are not as fast as a fully compiled TensorRT engine, but they offer a much better developer experience and support new models almost immediately.

Tomorrow I will look at NVIDIA Dynamo, which takes the disaggregation idea from day 18 and turns it into a concrete system for splitting prefill and decode across separate GPU pools.