Benchmarking was the art of measuring a system under controlled load. Observability is its sibling: the art of measuring a system in production, under real traffic, without breaking it.
Here's the uncomfortable truth I've learned: most inference outages aren't caused by model quality. They're caused by not seeing the problem until users do. Observability is the discipline of seeing problems before users do.
The three pillars
Observability has three pillars, and they answer three different questions:
- Metrics: "is something wrong?" Aggregated numbers over time: TTFT, TPS, queue depth, GPU utilization, KV cache hit rate.
- Logs: "what exactly happened?" Individual events: requests, errors, warnings, with context.
- Traces: "why did this specific request go wrong?" The full path of one request through every component.
You need all three. Metrics tell you the system is sick, logs tell you what the patient said, traces tell you which organ failed.
The metrics that matter for inference
Not all metrics are created equal. Here's the ones I actually watch, and why:
- TTFT (time to first token): the user's first impression. Under 300ms is good, under 1s is tolerable, over 2s is a problem. Spikes here usually mean prefill is backed up.
- TBT (time between tokens): the feel of the stream. Consistent TBT is more important than fast TBT. Jitter is the enemy.
- Queue depth: the leading indicator of overload. When it grows, you're about to have a bad time.
- KV cache hit rate: how often prefix caching works. High hit rate = low prefill cost = happy users.
- GPU utilization: useful, but misleading alone. A GPU at 90% could be thrashing on memory.
- Tokens per second: the output side of the ledger, and what you're billing.
Metrics are the dashboard of a car. Logs are the mechanic's notes. Traces are the black box. You wouldn't drive without a dashboard, and you shouldn't serve without one.
The stack
In practice, the modern inference observability stack looks like this:
- Prometheus scrapes metrics from the serving engine (vLLM, SGLang, Triton all expose them).
- Grafana turns them into dashboards you can actually read at 3am.
- OpenTelemetry traces a request from the API gateway through the engine to the GPU.
- Langfuse / Phoenix / LangSmith add the LLM-specific layer: prompt, completion, token usage, eval scores.
The key insight: instrument at every layer. Client, gateway, engine, GPU. A request that's slow could be slow anywhere, and if you only measure the engine, you'll blame the engine for a network problem.
The one metric to rule them all
If I had to pick one metric to watch, it's queue depth. Here's why: everything else is a lagging indicator. TTFT is already bad by the time you see it. Queue depth is the thing that predicts TTFT will be bad.
When queue depth grows, you have seconds to react before users feel it. That's the metric your autoscaler should watch, and the one your on-call should have pinned.
The takeaway
You can't fix what you can't see. And you can't see what you didn't instrument.
Tomorrow: the client code that talks to all of this, streaming and async.