There is no metric named `straggler`. There is only the gap between your best worker and your worst, and whether you're measuring it. In disaggregated serving, that gap is the difference between a p50 that looks great and a p99 that's killing your SLO.
Why disaggregation hides stragglers
In a disaggregated setup, prefill and decode run on different machines. A request's TTFT (time to first token) is set by the prefill path; its ITL (inter-token latency) is set by the decode path. If one decode worker is slow, every request routed to it pays the tax, and the p99 of the whole fleet follows that worker around.
The problem: a single slow worker is invisible in the aggregate. The p99 TTFT histogram doesn't tell you which worker is slow, or whether it's consistently slow or just having a bad moment.
The straggler score
There's no `straggler` metric in Dynamo. You derive it. The score is the gap between the worst worker and the median, per worker type:
max by (model) (dynamo_frontend_worker_last_time_to_first_token_seconds{worker_type="decode"})
-
quantile by (model) (0.5, dynamo_frontend_worker_last_time_to_first_token_seconds{worker_type="decode"})When this number is small, your fleet is uniform. When it spikes, you have a straggler. The score tells you that you have a problem; the worker table tells you who.
The worker load heatmap
Before the score, there's the load. Dynamo exposes per-worker load gauges: `worker_active_prefill_tokens` and `worker_active_decode_blocks`. These tell you whether a worker is busy because it's loaded or slow because it's sick.
- High load, high latency → the worker is saturated. Add capacity or rebalance.
- Low load, high latency → the worker is sick. Cordon and debug.
This is the same distinction as the EFA fabric counters: load vs health, and never conflate them.
The pipeline stages
Dynamo's frontend exposes per-stage request counts and durations: `preprocess`, `route`, `dispatch`, `accepted`, `received`, `transport_roundtrip`. The sum of `preprocess|route|dispatch` is roughly "waiting for first token".
- Router saturated → `route` stage grows.
- Prefill slow → `dispatch` stage grows.
- Decode backend slow → worker TTFT/ITL grows.
Stacking `dynamo_frontend_stage_requests` by stage × phase shows you where requests wait, which is the fastest way to localize a regression.
Router queues and KV
The KV-aware router is the heart of disaggregated serving. Its health is three numbers:
- Queue depth by worker type (`router_queue_pending_requests`) - is prefill or decode backing up?
- KV hit rate (`router_kv_hit_rate`) - how often the cache actually saves you a prefill.
- KV transfer estimated latency - the cost of moving KV from prefill to decode.
When the KV hit rate is high and transfer latency is low, disaggregation is paying for itself. When the transfer cost eats the prefill savings, it's not.
The dashboard that answers questions
Not charts, answers. The panels that matter, in order:
- SLO strip - TTFT/ITL/E2E p50-p99 vs targets, with error-budget burn.
- Pipeline stages - where requests wait.
- Straggler table - top-N workers by last TTFT/ITL.
- Worker load heatmap - loaded vs sick.
- Router queues + KV - is the router the bottleneck.
Everything else is Explore.
The takeaway
Disaggregated serving moves the bottleneck to the tail. Measure the gap between your best and worst worker, and you'll find the straggler before your users do.
Sources
- Dynamo metrics docs: the full metric taxonomy, including per-worker TTFT/ITL.
- Dynamo observability README: the system model and dashboard guidance.
- Dynamo repo: the disaggregated serving framework.
- DistServe paper: prefill-decode disaggregation and its latency model.