When people talk about compiling models with TensorRT, CUDA graphs, or Triton, they usually mean speed. But there's a second benefit that's just as valuable in production: security. Compiled kernels have a dramatically smaller attack surface than a Python framework.
The problem with Python in the serving path
PyTorch is enormous. It pulls in a Python interpreter, a JIT compiler, a huge dependency tree, and hundreds of packages, each of which can have CVEs. When you serve a model with PyTorch, every one of those CVEs is in your attack surface, and patching them means rebuilding and redeploying the whole stack.
It's not that PyTorch is uniquely insecure. It's that it's big, and big surfaces have more vulnerabilities. The math is simple: more code, more dependencies, more CVEs.
The compiled alternative
Port the model to a compiled runtime: TensorRT engine, CUDA graphs, Triton kernels, or a minimal C++ serving binary. What you get:
- No Python interpreter. The interpreter is a huge class of CVEs (arbitrary code execution, deserialization bugs). Remove it and you remove the whole class.
- Fewer dependencies. A compiled engine is a single artifact. No pip install, no transitive dependency tree, no supply-chain surprises.
- Smaller container. From gigabytes to hundreds of MB. Smaller image, smaller attack surface, faster cold start.
- Immutable artifact. The engine is built once and verified. No runtime codegen, no dynamic loading, no arbitrary Python at request time.
Serving with PyTorch is like running a server in a warehouse full of doors. Serving with a compiled engine is like running it in a room with one door. Same server, radically fewer ways in.
The numbers
It's not hand-wavy. A PyTorch serving container (torch + transformers + tokenizers + accelerate) ships 100-200 Python packages, and a CVE scan typically finds 50-150 vulnerabilities at any given moment. A compiled TensorRT engine served by a C++ runtime on a distroless base ships under 20 system libraries and usually single-digit CVEs. That's a 10-15x reduction in attack surface, before you even think about the interpreter.
The tradeoff
Compiling isn't free. Build time (minutes to hours), hardware specificity (an engine built for one GPU doesn't run on another), and less flexibility at runtime. But in production, where the model is fixed and the traffic is real, those costs are usually worth it.
This is why the frontier setups I run compile where it matters: the serving path is compiled, the control plane is Python, and the two never mix.
Sources
- NVIDIA TensorRT security considerations: the engine-as-trusted-artifact model.
- Google distroless containers: no shell, no package manager, single-digit CVEs.
- PyTorch security advisories: torch.load() pickle RCE, JIT bugs, the CVE accumulation.
- NVIDIA Triton Inference Server: C++ serving with TensorRT backend, no Python in the hot path.
The takeaway
Compiling a model is a performance win and a security win. Fewer CVEs, smaller containers, no interpreter in the hot path. The fastest path is often the safest one.