TensorRT-LLM: compile everything

When I first tried TensorRT-LLM, the build step took longer than training a small LoRA. I sat there wondering: why does inference need a compiler at all? PyTorch runs models just fine. The answer became obvious once I profiled the difference: a compiled engine on an H100 was pulling 30 to 40 percent more tokens per second than the same model in eager mode, at half the latency. Compilation is not a nice-to-have. It is where the last layer of performance lives.

What TensorRT-LLM actually is

TensorRT-LLM is NVIDIA's inference runtime specifically designed for large language models. It sits on top of the TensorRT compiler, the same one used for years in computer vision and speech, but wraps it with LLM-specific features: in-flight batching, paged KV caches, tensor parallelism, and quantization-aware kernels.

The core idea: take a model expressed in Python, convert it into a TensorRT engine, and execute that engine with a C++ runtime. The engine is a binary blob of fused CUDA kernels, custom memory layouts, and pre-computed execution plans. No Python overhead. No dynamic dispatch. Just raw kernel launches.

The build pipeline

The workflow has three stages:

• Define the model using TensorRT-LLM's Python API. This is not a direct PyTorch import. You describe the architecture using the library's own layer classes (like tensorrt_llm.models.LLaMAForCausalLM), which map to TensorRT operations.
• Build the engine with trtllm-build. This is where the compiler does its work: layer fusion, kernel selection, memory planning, and optional quantization calibration.
• Run inference using the C++ runtime or the Python tensorrt_llm.runtime wrapper.

A typical build command looks like this:

# Convert a Llama checkpoint to TRT-LLM format
python convert_checkpoint.py \
  --model_dir ./llama-3-8b \
  --output_dir ./tllm_checkpoint \
  --dtype float16

# Build the engine
trtllm-build \
  --checkpoint_dir ./tllm_checkpoint \
  --output_dir ./engine_dir \
  --gemm_plugin float16 \
  --max_batch_size 64 \
  --max_input_len 2048 \
  --max_seq_len 4096 \
  --paged_kv_cache enable

The --gemm_plugin flag is critical: it enables custom GEMM kernels instead of cuBLAS defaults, often delivering meaningful speedups for specific shapes. The --max_* flags are not suggestions; they are hard limits baked into the engine. If a request exceeds them, it fails. This is the trade-off: you give up dynamic shapes for deterministic performance.

What the compiler does under the hood

TensorRT's compiler performs several optimizations that are impossible in eager execution:

• Layer fusion: adjacent operations like linear + bias + activation get merged into a single kernel. Instead of three kernel launches (each with its own memory read/write), you get one. For a model with dozens of layers, this eliminates hundreds of kernel launch overheads per forward pass.
• Kernel auto-tuning: for each operation, the compiler benchmarks multiple kernel implementations (different tile sizes, different data layouts) and picks the fastest one for your specific GPU and tensor shapes. This is why the build step is slow: it is literally running a search.
• Memory planning: the compiler knows the full execution graph at compile time, so it can pre-allocate all intermediate tensors, reuse buffers, and minimize memory fragmentation. No runtime allocation, no garbage collection pauses.
• Precision calibration: when building with INT8 or FP8 quantization, the compiler can run calibration passes to determine optimal scaling factors per tensor, then bake those scales directly into the kernel parameters.

The LLM-specific features

Raw TensorRT could compile a transformer, but it would not know how to serve one efficiently. TensorRT-LLM adds the serving layer:

• In-flight batching: new requests join a running batch without waiting for the current batch to finish. This is the same idea as vLLM's continuous batching, implemented at the C++ runtime level.
• Paged KV cache: the KV cache is split into fixed-size blocks allocated on demand, just like PagedAttention in vLLM. No over-allocation, no fragmentation.
• Tensor parallelism: the engine can be built for multi-GPU execution with NCCL communication baked into the graph. You specify --tp_size 4 at build time and the compiler inserts all-reduce operations at the right points.
• Custom attention kernels: TensorRT-LLM ships XQA (cross-query attention) kernels optimized for GQA and MQA patterns common in modern architectures like Llama 3. These are hand-written CUDA, not auto-generated.

Where it hurts

Compilation is not free. The downsides are real:

• Build times: building an engine for a 70B model with multiple quantization profiles can take 30 minutes to over an hour. Every change to batch size, sequence length, or precision requires a rebuild.
• Shape rigidity: the max_batch_size and max_seq_len you pick at build time are your hard ceilings. Pick too low and you reject requests. Pick too high and you waste memory on pre-allocated buffers.
• Model support lag: when a new architecture drops (say, a new attention variant or a novel MoE routing scheme), you have to wait for TensorRT-LLM to add support. With vLLM or SGLang, you can often run the model in eager PyTorch on day one.
• Debugging opacity: when something goes wrong inside a compiled engine, you cannot insert print statements. You are debugging a binary blob.

TensorRT-LLM vs. vLLM and SGLang

The mental model I use: vLLM and SGLang are serving frameworks that happen to optimize execution. TensorRT-LLM is a compiler that happens to include a serving runtime. They solve the problem from opposite ends.

In practice, TensorRT-LLM engines often deliver the highest single-stream throughput and lowest per-token latency, especially on NVIDIA hardware with FP8 quantization. But vLLM and SGLang offer faster iteration, broader model support, and easier deployment. Many production setups actually use TensorRT-LLM engines as a backend within a serving framework, getting the best of both worlds.

If you need the absolute fastest inference on NVIDIA GPUs and you can afford the build step, compile. If you need flexibility and fast iteration, stay eager. If you are at scale, you will probably end up doing both.

Tomorrow: NVIDIA Dynamo, the orchestration layer that takes disaggregated serving to the next level by splitting prefill and decode across separate GPU pools.