The first time I put torch.compile in front of an LLM inference server, it spent the better part of a minute compiling and handed back a 6% speedup. I nearly wrote the feature off. The next morning, on a different model, it compiled in twelve seconds and ran more than twice as fast. Same GPU, same driver, same PyTorch build. The difference had nothing to do with FLOPs or memory bandwidth. It was graph breaks, the points where the compiler gave up on my Python and dumped execution back into the interpreter.
That is the whole game, and it is worth stating plainly: torch.compile is not a compiler in the sense that gcc or nvcc is. It does not read your source and emit a self-contained binary. It watches your Python run, lifts the stretches it can handle into a graph, compiles those, and stitches the compiled regions back together with eager fallbacks for everything it could not capture. Your speedup is a function of one ratio: how much of the hot path landed in compiled regions versus how much leaked back into the interpreter.
Three components do the work:
- TorchDynamo captures the graph.
- AOTAutograd traces the backward pass ahead of time so it can be compiled alongside the forward.
- TorchInductor generates code. Inductor is the default backend, but Dynamo is deliberately pluggable: pass backend="openxla" to target TPUs, or a vendor backend like IPEX for Intel.
One detail trips up everyone at least once: nothing compiles when you call torch.compile(model). Compilation is lazy. It fires on the first forward pass that reaches a given code path, and the artifacts are cached on disk. So the "why is my first step so slow" question has a boring answer, and the "why did step 900 suddenly stall" question is the interesting one.
Dynamo is the part that decides whether any of this works
Dynamo is the only piece of this stack you need to understand deeply. Everything downstream is competent and mostly invisible when it works. Dynamo is where your code either compiles or doesn't, and it tends to fail quietly: no exception, just a slow model and a log line you were not watching.
Older approaches like TorchScript tried to solve capture by forcing you into a restricted Python subset: no data-dependent control flow, no third-party libraries, no exotic objects. It failed in production for the obvious reason. Real model code is full of exactly those things. Dynamo took the opposite bet. Instead of constraining the language, it hooks CPython's frame evaluation API (PEP 523, the extension point that lets an external tool replace the default bytecode evaluator) and intercepts frames as they execute.
At the C layer, Dynamo installs its own frame evaluator through CPython's private hook, _PyInterpreterState_SetEvalFrameFunc. When CPython is about to run a Python function, it has already built a PyFrameObject holding the bytecode, the locals, and the value stack. Rather than let _PyEval_EvalFrameDefault run that frame, Dynamo takes it and walks the bytecode itself, one instruction at a time, through a symbolic interpreter called InstructionTranslator.
The mental model that matters: as Dynamo walks the bytecode, it is running your function symbolically rather than concretely. LOAD_FAST and STORE_FAST shuffle references on a symbolic stack. When it hits a binary op whose operands are tensors, it does not multiply anything, it appends a node to an FX graph and keeps going. Tensors are tracked as TensorVariable handles carrying shape, dtype, and device. Dynamic dimensions are tracked as symbolic integers backed by sympy expressions, so a size can stay s0 * 2 + 16 instead of collapsing to a concrete 1024 at trace time. That symbolic-shape machinery is the difference between a graph that generalizes across batch sizes and one that recompiles every time an input changes.
Where it gets interesting is control flow. A branch on a static Python value, a config flag, a constant, is resolved at trace time; Dynamo follows the live branch and never records the dead one. But a branch on a runtime tensor value, the classic if x.sum() > 0:, cannot be resolved symbolically. Dynamo does not know which way it goes. So it stops.
Graph breaks, and why I have spent more time on them than anything else
When Dynamo hits something it cannot trace through, a data-dependent branch, a print, a call into a C extension it cannot see inside, it takes a graph break. It compiles everything accumulated so far into one subgraph, falls back to the eager interpreter for the offending instruction, and tries to resume tracing on the other side.
@torch.compile
def f(x):
y = x * 2 # 1. traced into subgraph 0
z = y + 1 # 2. still subgraph 0
print(z.shape) # 3. graph break: a side effect Dynamo won't capture
w = z.relu() # 4. traced into subgraph 1
return wTwo subgraphs, one print between them. Each subgraph compiles and optimizes in isolation. You have lost every fusion opportunity that would have crossed the boundary, and you pay an extra handoff between compiled code and the interpreter on every call. torch._dynamo.explain(f)(x) will lay it out for you.
torch.compile is a highway that runs through your Python. Where the road is clear, you fly at 2x. Every graph break is a toll plaza where you slow to a stop, merge back onto city streets, and re-enter the highway on the other side. The number of toll plazas on your hot path is the whole game.
Why this matters for inference
For LLM serving, the decode path is where torch.compile pays off most: the autoregressive loop is a tight sequence of small ops where eager-mode Python overhead and kernel launch latencies dominate. Capturing the model graph via torch.compile and CUDA graphs eliminates that overhead. But the payoff is entirely conditional on graph breaks, and the decode loop is full of them: data-dependent sampling, KV cache indexing, dynamic shapes.
The lesson I keep relearning: your speedup is not a property of the model, it is a property of how cleanly your code traces. Same GPU, same driver, same PyTorch build, wildly different results depending on where the graph breaks land.
torch.compile is not a compiler. It is a graph breaker with a compiler attached, and the graph breaks decide everything.