Reusable benchmark harness

I used to benchmark LLM inference servers by writing a quick script each time, hardcoding the prompt, the concurrency, and the metrics I cared about. Every new experiment meant copying the previous script, tweaking the parameters, and inevitably breaking something. The results were in different formats, making comparison painful.

So I sat down and built a proper harness. Nothing fancy, just a Python tool that takes a YAML config, runs a load test against an OpenAI-compatible endpoint, streams back the responses, and computes the metrics that actually matter. I have used it for every benchmark since, and it has been one of the highest-leverage things I have built.

What the harness measures

The metrics that matter for LLM inference benchmarking are different from typical web service benchmarks. HTTP status codes and response times are not enough. You need:

• Time to first token (TTFT): time from request send to the first SSE data event. This captures prefill latency plus queueing.
• Inter-token latency (ITL): time between consecutive SSE events. This captures decode speed. I report the median, P90, and P99.
• End-to-end latency: time from request send to final SSE event. The user-visible number.
• Throughput: total output tokens per second across all concurrent requests. This is the number you care about for capacity planning.
• Output token count: verify the model actually generated the expected number of tokens. A model that stops early inflates throughput numbers.

The harness records timestamps at every SSE event, computes these metrics per request, then aggregates across the full run.

The core architecture

The harness uses asyncio with aiohttp for concurrent requests. Each request is an async task that opens an SSE stream, records timestamps for every chunk, and stores the raw timing data. After all requests complete, a separate reporting module computes percentiles and writes CSV.

import asyncio
import aiohttp
import time
import json
from dataclasses import dataclass, field

@dataclass
class RequestResult:
    request_id: int
    prompt_tokens: int
    output_tokens: int
    ttft_ms: float
    itl_ms: list = field(default_factory=list)
    end_to_end_ms: float = 0.0
    error: str = ""

async def send_request(session, url, payload, req_id):
    result = RequestResult(request_id=req_id,
                           prompt_tokens=payload.get("max_tokens", 0),
                           output_tokens=0,
                           ttft_ms=0.0)
    t_start = time.perf_counter()
    t_prev = t_start

    try:
        async with session.post(url, json=payload) as resp:
            async for line in resp.content:
                decoded = line.decode().strip()
                if not decoded.startswith("data:"):
                    continue
                if decoded == "data: [DONE]":
                    break

                t_now = time.perf_counter()
                chunk = json.loads(decoded[5:])
                delta = chunk["choices"][0].get("delta", {})

                if delta.get("content"):
                    result.output_tokens += 1
                    if result.output_tokens == 1:
                        result.ttft_ms = (t_now - t_start) * 1000
                    else:
                        result.itl_ms.append((t_now - t_prev) * 1000)
                    t_prev = t_now

    except Exception as e:
        result.error = str(e)

    result.end_to_end_ms = (time.perf_counter() - t_start) * 1000
    return result

The key detail is using time.perf_counter() for all timing. It has sub-microsecond resolution on Linux and is monotonic, so it will not jump if the system clock adjusts. Never use time.time() for benchmarking.

Request generation

A good benchmark needs realistic request shapes. I define prompts by their token count, not by text content, because what matters for performance is the computation, not the semantics. The harness supports three modes:

• Fixed: all requests have the same prompt length and max output tokens. Useful for controlled experiments.
• Uniform random: prompt length and output length are drawn uniformly from configured ranges. Good for stress testing.
• Trace replay: read a CSV of real request shapes (prompt_tokens, output_tokens) captured from production logs. The gold standard for capacity planning.

# Config example (YAML)
endpoint: "http://localhost:8000/v1/chat/completions"
model: "meta-llama/Llama-3.1-8B-Instruct"
concurrency: [1, 4, 8, 16, 32]
duration_seconds: 120
warmup_seconds: 10

request_shape:
  mode: "uniform"
  prompt_tokens_range: [128, 2048]
  output_tokens_range: [64, 512]

# For trace replay:
# request_shape:
#   mode: "trace"
#   trace_file: "production_shapes.csv"

For synthetic prompts, I fill the prompt with repeated tokens to reach the target length. The specific tokens do not affect inference performance (the model does the same compute regardless of content), so using a simple filler like repeating "hello " works fine.

Running sweeps

The most valuable feature is the concurrency sweep. Instead of running one test at a fixed concurrency, the harness iterates through a list of concurrency levels and runs each one for a configurable duration. This produces the throughput-vs-latency curve that tells you where your server saturates.

async def run_sweep(config):
    results_by_concurrency = {}

    for concurrency in config["concurrency"]:
        print(f"Running concurrency={concurrency}...")
        results = await run_at_concurrency(
            config["endpoint"],
            config["model"],
            concurrency,
            config["duration_seconds"],
            config["request_shape"]
        )
        results_by_concurrency[concurrency] = results

    return results_by_concurrency

The output is a CSV with one row per concurrency level, reporting median TTFT, P99 TTFT, median ITL, P99 ITL, throughput (tokens/s), and the number of completed requests. I load this into a notebook or plot it directly to find the saturation point.

Avoiding common benchmarking mistakes

Building the harness forced me to think carefully about benchmarking methodology. Some mistakes I have made and now guard against:

• Coordinated omission: if your harness waits for a response before sending the next request, you are measuring throughput under zero load, not under realistic concurrency. Use a fixed number of concurrent coroutines that each send a new request as soon as their previous one completes. This maintains a constant number of in-flight requests.
• Ignoring output length variance: two runs with different average output lengths are not comparable for throughput. Always control or record the output length and normalize when comparing.
• Not measuring the right percentile: median latency tells you the typical experience. P99 tells you the worst-case. For SLA compliance, P99 is what matters. I report both, always.
• Benchmarking a cold server: if the server just started and the CUDA context is not initialized, the first batch of requests will be much slower. Warmup is not optional.

Output and reporting

The harness writes two files: a detailed CSV with per-request metrics (for deep analysis) and a summary CSV with per-concurrency aggregates (for quick comparison). The summary looks like this:

concurrency,requests,median_ttft_ms,p99_ttft_ms,median_itl_ms,p99_itl_ms,throughput_tok_s
1,45,28.3,35.1,11.2,14.8,87.4
4,178,32.1,48.7,11.5,16.2,341.2
8,340,41.2,78.3,12.1,19.4,652.8
16,612,68.7,142.5,13.8,28.6,1124.3
32,980,142.3,387.2,18.4,52.1,1687.5

From this you can immediately see the tradeoff: throughput increases with concurrency but latency degrades. The sweet spot for this server is around concurrency 8 to 16, where throughput is high but P99 TTFT is still under 150 ms.

A benchmark you cannot reproduce is a benchmark you cannot trust. Pin the model version, the server config, the request shapes, and the hardware. Record all of these alongside the results. Future you will thank present you.

For load testing with an existing tool, see load test with Locust. For profiling the server side with Nsight, see profile with Nsight Systems. For the async client that this harness builds on, check async batch client with asyncio.