Metrics covered the "is something wrong" signal. But metrics have a blind spot: they aggregate. When a single request is slow, metrics can't tell you why, because the slow request is averaged away.
That's what traces are for. Today I traced one request through the whole stack, and it changed how I debug.
The anatomy of a trace
A trace is a tree of spans. Each span is one unit of work with a start, an end, and a set of attributes. The root span is the whole request. Child spans are the components it passed through.
For an LLM request, the trace looks like this:
POST /v1/chat/completions (root span)
├── auth (50ms)
├── router (30ms)
├── engine: prefill (400ms)
│ ├── tokenize (2ms)
│ ├── forward pass (350ms)
│ └── sample (5ms)
├── engine: decode x 200 (each ~8ms)
│ ├── forward pass (7ms)
│ └── sample (1ms)
└── response serialization (10ms)With this, "why is it slow" becomes "which span is fat". Not a guess, a measurement.
How it works
OpenTelemetry is the standard. It's a set of APIs and SDKs that emit spans, plus a collector that receives them and forwards to a backend (Jaeger, Tempo, SigNoz, whatever). The magic is context propagation: a trace ID travels with the request, and every component that sees it adds its own span to the same trace.
The propagation is what makes it work. Your API gateway starts a trace, passes the trace ID in a header, the engine picks it up and adds spans, the GPU exporter adds its own. One ID, one tree, the whole request.
What I learned tracing real requests
- The engine is rarely the whole story. I've seen "slow inference" turn out to be a slow auth call, a misconfigured router, or a client that wasn't reading the stream.
- Prefill dominates TTFT. For a long prompt, the prefill span is the fat one. That's the roofline again: prefill is compute-bound, so it scales with prompt length.
- Decode spans are uniform. When they aren't, you have a scheduling problem, not a model problem.
- Streaming changes the math. With SSE, the client sees the first token before the request is done. You have to trace the stream, not just the request.
Metrics are the smoke detector. Traces are the fire inspector. The smoke detector says "fire", the inspector walks through the building and finds the exact faulty wire.
The takeaway
If you only measure aggregates, every slow request looks like everyone else's. Traces are how you find the one that's different, and the one that's different is the one that's broken.
Tomorrow: load testing with Locust, the controlled chaos that finds the saturation point.