Profiling is the discipline I keep coming back to in this notebook. The roofline tells you where the ceiling is. Nsight Systems tells you where the time actually goes. But there's a gap between "I can see the problem" and "I have fixed the problem", and that gap is where the hours go.
So I built agents that close it.
The loop, automated
The manual loop is: profile, read the trace, form a hypothesis, change one thing, re-profile, repeat. The agentic loop is the same, but the reading and the hypothesizing are automated.
Nsight Systems, read by a machine
nsys profile produces a trace file that is, frankly, too much data for a human to read line by line. That's exactly the kind of thing an agent is good at: take the trace, find the long gaps, correlate them with the kernels, and report back in one sentence.
The agent doesn't replace the engineer. It replaces the reading. The engineer still decides what to change. But the loop goes from "spend an hour reading a trace" to "spend a minute reading a summary".
KV-aware routing
The other thing the agents do is build the KV-aware router. In a disaggregated setup, the router decides which decode GPU gets a request. The naive router spreads load evenly. The smart router asks: which GPU already has this prefix in its cache? and routes there, avoiding a re-prefill entirely.
This is the cache-aware scheduling that SGLang's RadixAttention and NVIDIA Dynamo both enable. The router keeps a map of what's cached where, updates it as conversations progress, and uses it to make routing decisions. The hit rate on the cache pool is the metric that matters.
Dynamo as the skeleton
NVIDIA Dynamo is the framework that ties it together: it lets you build disaggregated serving systems where prefill and decode are separate services, connected by a fast transport. It's the skeleton our agents configure and tune. The agents profile the Dynamo graph, find the hot path, and adjust the batching or the routing.
This is the frontier of the field: not just serving models, but serving them with software that watches itself and improves itself. Dynamo's Planner is the closest thing to a production agent today: it monitors GPU capacity and TTFT/ITL SLOs, then decides whether to serve disaggregated or colocated, shifts GPUs between prefill and decode, and adds capacity to the bottlenecked phase. AIConfigurator goes further, simulating 10,000+ deployment configs in seconds to find the Pareto-optimal topology before you burn a single GPU-hour.
Sources
- NVIDIA Dynamo KV-aware router docs: routes by
cost = new_blocks × prefill_weight + active_blocks × decode_weight. - SGLang RadixAttention blog: the radix tree that all KV-aware routing builds on.
- Nsight Systems user guide:
nsys profile, NVTX ranges, focused profiling. - Introducing NVIDIA Dynamo: Planner, Smart Router, NIXL, 30x on DeepSeek-R1.
- Baseten: 2x TTFT with KV-aware routing: production validation on Qwen3-Coder 480B.
- Dynamo AIConfigurator: simulates 10K+ deployment configs in seconds.
- vLLM profiling:
VLLM_TORCH_PROFILER_DIRand nsys integration.
The takeaway
The best profiler is the one that runs without you. The best router is the one that knows what's cached. The best stack is the one that tunes itself.