Emit Prometheus metrics from an inference server

Observability theory was covered earlier. This is the practice: actually getting Prometheus metrics out of a serving engine and understanding what they mean.

The good news: vLLM, SGLang, and Triton all ship Prometheus endpoints out of the box. You don't have to build instrumentation from scratch. You have to know which numbers matter and what they're telling you.

The endpoint

vLLM exposes metrics at /metrics in Prometheus format. SGLang does too. The format is dead simple, text-based, and designed to be scraped:

# HELP vllm:num_requests_running The number of requests currently running
# TYPE vllm:num_requests_running gauge
vllm:num_requests_running 3

# HELP vllm:time_to_first_token_seconds ...
# TYPE vllm:time_to_first_token_seconds histogram
vllm:time_to_first_token_seconds_bucket{le="0.1"} 12

That's it. A gauge is a single number (current queue depth). A counter only goes up (total requests). A histogram buckets values (TTFT distribution). Prometheus scrapes this every 15 seconds and stores it.

The metrics I actually query

Here's my PromQL cheat sheet, the queries I run when something feels wrong:

• Queue depth: vllm:num_requests_waiting: the leading indicator. If this is climbing, you're about to be slow.
• TTFT p95: histogram_quantile(0.95, rate(vllm:time_to_first_token_seconds_bucket[5m])): how the first token actually feels.
• TBT p95: histogram_quantile(0.95, rate(vllm:time_per_output_token_seconds_bucket[5m])): the feel of the stream.
• GPU utilization: DCGM_FI_DEV_GPU_UTIL from the DCGM exporter: but pair it with memory to avoid the "90% util, still slow" trap.
• KV cache usage: vllm:gpu_cache_usage_perc: when this hits 100%, requests start getting evicted or rejected.

The trap: dashboards nobody reads

Here's what I learned the hard way: a dashboard with 40 panels is a dashboard nobody reads. A dashboard with 5 panels that answer "is it slow, and why" is a dashboard that saves your weekend.

My rule: every panel must answer a question you'd ask at 3am. "Is it slow?" (TTFT/TBT). "Is it overloaded?" (queue depth, cache usage). "Is it the GPU?" (utilization, memory). "Is it the network?" (client-side latency, error rate). Five panels, five questions.

The takeaway

Your engine already speaks Prometheus. The only question is whether you're listening.

Tomorrow: turning those metrics into a Grafana dashboard you can actually read.