Benchmarking: tooling and profiling

I've seen teams benchmark their inference stack by running 10 requests sequentially and reporting the average latency. That number is almost meaningless. It tells you nothing about how the system behaves under concurrent load, nothing about tail latency, and nothing about throughput. Real benchmarking is about understanding the system's behavior across its operating range, and it requires both the right tools and the right methodology.

What to measure

LLM inference has more performance dimensions than a typical web service. The key metrics:

• Time to first token (TTFT). The latency from request arrival to the first output token. This is dominated by prefill time and queueing delay. Users perceive this as "how long until it starts typing."
• Inter-token latency (ITL). The time between consecutive output tokens during decode. This determines the perceived streaming speed. Typically 10 to 50 ms per token on modern hardware.
• End-to-end latency. Total time from request to last token. This is TTFT plus (number of output tokens times ITL).
• Throughput. Tokens per second across all concurrent requests. This is what you're optimizing when you care about cost efficiency.
• Goodput. Throughput for requests that actually met your SLO. If your SLO is "p99 TTFT under 2 seconds" and 5% of requests miss it, your goodput is lower than your throughput.

Always report percentiles (p50, p90, p99), not averages. Average latency hides the tail, and the tail is where users feel pain.

The tools

Several open-source tools have emerged specifically for LLM inference benchmarking:

GenAI-Perf (NVIDIA). Part of the Triton Inference Server ecosystem. It generates load against an OpenAI-compatible API and reports TTFT, ITL, throughput, and latency percentiles. It understands streaming responses and can parse SSE token streams. This is my default for quick benchmarks.

# GenAI-Perf: benchmark an OpenAI-compatible endpoint
genai-perf profile \
  -m llama-3.1-70b \
  --endpoint-type chat \
  --url http://localhost:8000 \
  --streaming \
  --concurrency 32 \
  --input-tokens-mean 512 \
  --output-tokens-mean 256 \
  --measurement-interval 60000

vLLM's benchmark_serving.py. Ships with vLLM and is tightly integrated with its metrics. It supports configurable request rates (Poisson arrival), input/output length distributions, and multi-model scenarios. The output includes TTFT and ITL distributions.

# vLLM's built-in benchmark
python benchmark_serving.py \
  --backend openai \
  --base-url http://localhost:8000 \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --dataset-name sharegpt \
  --request-rate 10 \
  --num-prompts 500

Locust. Not LLM-specific, but I use it when I need custom load patterns or want to simulate realistic user behavior (e.g., users who read tokens as they stream and then send a follow-up). Locust's Python-based test scripts make it easy to write LLM-aware load generators.

Nsight Systems. For GPU-level profiling. When you need to know why a kernel is slow, not just that it is slow. Nsight traces CUDA kernels, memory copies, and CPU activity on a timeline. It's the difference between "inference is slow" and "the attention kernel takes 3.2 ms because it's reading 48 MB of KV cache from HBM."

Methodology: how to not fool yourself

The most common benchmarking mistakes I've seen:

• No warmup. The first few requests trigger CUDA kernel compilation (torch.compile, TensorRT engine build) and memory allocation. Always run 50 to 100 warmup requests before measuring. Discard them from your results.
• Fixed concurrency instead of fixed request rate. Running "32 concurrent requests" means the system is always saturated. Real traffic has a request rate with natural variation. Use Poisson arrivals (requests arrive at random intervals averaging N per second) to simulate realistic conditions. vLLM's benchmark script supports --request-rate for this.
• Uniform input lengths. Real traffic has a distribution of prompt lengths. A benchmark with all-512-token prompts hides the fact that a single 8192-token prompt might blow out your KV cache and evict other requests. Use a dataset like ShareGPT that has natural length variation.
• Ignoring queuing delay. TTFT includes time spent waiting in the serving engine's queue. If you measure TTFT only at low concurrency, you'll never see the queuing component. Sweep concurrency from 1 to saturation to understand where queuing kicks in.

Profiling: going deeper

Benchmarking tells you what is slow. Profiling tells you why. For LLM inference, the profiling stack looks like:

• torch.profiler. Traces PyTorch operations, CUDA kernels, and memory allocations. Exports to Chrome trace format or TensorBoard. Good for understanding which layers dominate inference time.
• Nsight Systems (nsys). NVIDIA's system-level profiler. Shows GPU utilization, kernel execution, memory copies, and CPU-GPU synchronization points on a timeline. Essential for diagnosing pipeline bubbles.
• Nsight Compute (ncu). Kernel-level profiler. Shows occupancy, memory throughput, warp stalls, and instruction mix for individual CUDA kernels. Use this when you know which kernel is slow and want to know why.

# Profile with torch.profiler
import torch
from torch.profiler import profile, ProfilerActivity

with profile(
    activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
    schedule=torch.profiler.schedule(wait=2, warmup=3, active=5),
    on_trace_ready=torch.profiler.tensorboard_trace_handler("./logs"),
    record_shapes=True,
    profile_memory=True,
    with_stack=True,
) as prof:
    for step in range(10):
        output = model.generate(input_ids, max_new_tokens=64)
        prof.step()

What good numbers look like

As a rough calibration for a 70B model on a single 8xH100 node in FP16 with tensor parallelism across all 8 GPUs:

• TTFT at batch size 1: 200 to 400 ms for a 512-token prompt.
• ITL at batch size 1: 20 to 30 ms per token.
• Throughput at high concurrency: 2000 to 4000 output tokens per second across all requests.
• Saturation point: typically 64 to 128 concurrent requests before TTFT p99 exceeds 5 seconds.

These numbers shift significantly with quantization (INT8 roughly doubles throughput), KV cache compression, speculative decoding, and the specific serving engine. The point is not to memorize them but to have a baseline so you can tell when something is wrong.

Benchmarking is a skill, not a task. The tools are straightforward; the discipline of controlling variables, reporting distributions, and questioning your own results is what separates useful benchmarks from misleading ones.

Next: client code for streaming and async, because the server is only half the story.