Skand Tandon
Production Systems

End-to-end latency breakdown

The model forward pass is only part of the story. Tokenization, scheduling, network hops, detokenization, and SSE framing all add up. I traced a request from client send to final token received and broke down every millisecond.

2025-12-16 production systems 8 min read

When someone asks "what is the latency of your model?", the answer is almost always incomplete. They usually mean time-to-first-token or inter-token latency, both of which describe the model's forward pass. But the user does not experience the forward pass. The user experiences everything from the moment they hit send to the moment the last character appears on screen. And that end-to-end number is always larger, sometimes significantly, than the model-only number.

I spent a day instrumenting every stage of a typical LLM serving stack, from client to load balancer to inference server and back, and the breakdown was eye-opening. Here is what I found.

The full request lifecycle

A typical production LLM request traverses these stages:

That is at least 12 distinct stages for the first token alone. Let me quantify each one.

The numbers, stage by stage

Here is what I measured on a setup with a 7B model on a single H100, serving through FastAPI + uvicorn behind an nginx load balancer, with the client on the same network (sub-millisecond RTT):

Stage                          Typical (ms)
---------------------------------------------
Client JSON serialization         0.1 - 0.5
Network hop (same region)         0.5 - 2.0
Load balancer (nginx)             0.2 - 1.0
HTTP parsing (uvicorn)            0.3 - 1.0
Tokenization (512 tokens)         1.0 - 3.0
Queue wait (varies wildly)        0.0 - 500+
Prefill (512 tokens, 7B)         15.0 - 25.0
First decode step                  8.0 - 12.0
Detokenization                     0.1 - 0.3
SSE framing                        0.1 - 0.2
Network return                     0.5 - 2.0
---------------------------------------------
Total TTFT (no queue)             26 - 47 ms
Total TTFT (with queue)           26 - 547+ ms

The model forward pass (prefill + first decode) is typically 23 to 37 ms for this setup. The non-model overhead adds 3 to 10 ms. That is a 10 to 30 percent overhead when there is no queueing. Acceptable, but not negligible.

The real story, though, is the queue wait. Under load, queue wait can dwarf everything else. When the server is at capacity and all batch slots are full, a new request has to wait for a slot to open. This is where TTFT goes from 30 ms to 500 ms or more, and it has nothing to do with the model.

Tokenization: the hidden tax

Tokenization surprised me. For short prompts (under 100 tokens), it is sub-millisecond and irrelevant. But for long prompts, it adds up. Tokenizing 8K tokens with the Llama tokenizer takes 5 to 8 ms on a single CPU core. That is 20 to 30 percent of the prefill time for a 7B model.

This matters because tokenization happens on the CPU, in the Python event loop, and it is synchronous in most serving frameworks. While the CPU is tokenizing one request, the event loop is blocked and cannot process incoming SSE responses for other requests. In high-throughput deployments, I have seen tokenization of long prompts cause visible stalls in the SSE stream of concurrent requests.

# Tokenization benchmark
from transformers import AutoTokenizer
import time

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
text = "word " * 8000  # ~8K tokens

start = time.perf_counter()
tokens = tokenizer.encode(text)
elapsed = (time.perf_counter() - start) * 1000
print(f"{len(tokens)} tokens in {elapsed:.1f} ms")
# Typical output: 8001 tokens in 6.2 ms

The fix is to offload tokenization to a separate thread or process. vLLM does this with its tokenizer pool (--tokenizer-pool-size), which runs tokenization in background processes. SGLang handles it similarly. If you are building your own serving layer, do not tokenize on the main event loop.

Watch out

Detokenization has a similar issue but smaller magnitude, because it processes one token at a time. However, incremental detokenization with SentencePiece-based tokenizers requires maintaining state across tokens to handle multi-byte characters correctly. A naive implementation that decodes each token independently will produce garbled output for non-ASCII text.

Queue wait: the silent killer

Queue wait is the most variable component and the hardest to optimize. It depends on three things: the arrival rate of requests, the batch size the server can handle, and the average generation length (which determines how long each batch slot is occupied).

With continuous batching, a slot opens as soon as one request in the batch finishes generating, rather than waiting for the entire batch to complete. This dramatically reduces queue wait compared to static batching, where a request might wait for the slowest request in the batch to finish before the next batch starts.

But even with continuous batching, if the arrival rate exceeds the throughput, the queue grows without bound. I monitor two metrics to catch this:

Network and SSE: death by a thousand frames

For streaming responses, each token generates a separate SSE event. At 50 tokens per second, that is 50 HTTP frames per second per request. With 100 concurrent requests, the server is pushing 5000 frames per second. The per-frame overhead of SSE is small (a few bytes of framing), but the kernel has to context-switch for each send call, and TCP Nagle's algorithm can add up to 40 ms of buffering if not disabled.

Practical tips I have learned:

Cross-region: when the network dominates

Everything changes when the client is in a different region from the inference server. A cross-region network hop adds 50 to 200 ms of round-trip time. For TTFT, this adds one RTT (the request must reach the server). For inter-token latency perceived by the client, it adds one RTT per SSE chunk.

In practice, SSE chunks are pipelined on a persistent connection, so the inter-token overhead is not a full RTT per token. But the first-byte latency absolutely includes the full RTT. If your inference server is in us-east and your user is in ap-south, you are adding 200 ms to TTFT before the model even starts running. This is why geo-aware routing matters for latency-sensitive applications.

If you are only measuring model latency, you are measuring the wrong thing. Instrument the full path, from client send to client receive, and break it down by stage. The optimization that gives you the biggest latency win might not be in the model at all.

For how to build the streaming client, see SSE streaming client. For queue management with batch formation, see priority request queue. For load balancing strategies, see round-robin and least-connections load balancers.