Skand Tandon
Production Systems

Grafana dashboard: TTFT, TBT, queue depth

The five-panel dashboard that answers every 3am question. Built from real Prometheus queries, not vibes.

2026-02-17 production systems 7 min read

Yesterday I said the whole art of observability is choosing what to record. Today I'm showing my work: the exact dashboard I built, panel by panel, and the reasoning behind each one.

This isn't a 40-panel monster. It's five panels, and each one answers a question you'd ask at 3am.

Panel 1: Is it slow?

TTFT and TBT, p50 and p95, over the last hour. Two time series, one panel. When a user says "it feels slow", this is the first place you look.

histogram_quantile(0.95, rate(vllm:time_to_first_token_seconds_bucket[5m]))
histogram_quantile(0.50, rate(vllm:time_to_first_token_seconds_bucket[5m]))
histogram_quantile(0.95, rate(vllm:time_per_output_token_seconds_bucket[5m]))

If TTFT is high but TBT is fine, it's a prefill problem. If both are high, it's overload. If TBT is jittery, it's scheduling or contention.

Panel 2: Is it overloaded?

Queue depth and KV cache usage, stacked. Queue depth is the leading indicator. KV cache usage is the hard ceiling: when it hits 100%, requests get evicted or rejected.

vllm:num_requests_waiting
vllm:gpu_cache_usage_perc * 100

If queue depth is climbing but cache usage is fine, you have a scheduling problem. If cache usage is pinned at 100%, you have a memory problem.

Panel 3: Is it the GPU?

GPU utilization and memory, from the DCGM exporter. The trap here is pairing them: 90% util with 30% memory means compute-bound. 90% util with 95% memory means you're about to OOM. Same number, different diagnosis.

DCGM_FI_DEV_GPU_UTIL
DCGM_FI_DEV_MEM_COPY_UTIL

Panel 4: Is it the network?

Client-side latency and error rate. This is the panel that saves you from blaming the GPU for a network problem. If the engine is fast but users are slow, it's not the engine.

rate(http_requests_total{status=~"5.."}[5m])
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

Panel 5: Is it the model?

Token throughput and KV cache hit rate. This is the "is the model doing what we expect" panel. A sudden drop in throughput with steady utilization usually means the model got slower, not the system.

rate(vllm:generation_tokens_total[5m])
vllm:prefix_cache_hit_rate
Mental model

A good dashboard is like a pilot's instrument panel: five gauges, each with a red zone, and a trained eye that knows which gauge to trust when they disagree.

The workflow

Here's how the dashboard actually gets used in a real incident:

  1. Alerts page at 2am: TTFT p95 over 2s.
  2. Panel 1 confirms: TTFT is high, TBT is fine.
  3. Panel 2: queue depth is climbing, cache is fine.
  4. Panel 3: GPU util is 95%, memory is 40%.
  5. Diagnosis: a traffic spike is overloading prefill. The autoscaler is reacting, but it needs 3 minutes to boot a replica.
  6. Action: nothing to fix, it's a transient spike. But now you know to add a warm replica for this traffic pattern.

Five panels, one diagnosis, no guessing.

The takeaway

A dashboard isn't a collection of charts. It's a decision tool. If a panel doesn't help you decide, it doesn't belong.

Tomorrow: tracing a single request through the whole stack with OpenTelemetry.