Methodology

Every number on this site is somebody else's.

That bothers me. The frontier pages cite vendors and Artificial Analysis because that is what exists publicly, and none of it was measured on the hardware I actually care about. This page is the harness I would run to fix that: the exact configuration, the load pattern, what to record, and the mistakes that quietly make a result worthless.

What counts as a result

before touching a flag

A throughput number with no latency constraint is not a result, it is a press release. Any serving benchmark has to pin four things or it cannot be compared to anything, including itself a week later.

An interactivity target

Throughput and per-user latency trade against each other continuously. "17,000 tokens per second" means nothing without saying at what time-per-output-token. I would fix a target, say 50 ms between tokens, and report the throughput achievable while holding it at p95.

A fixed input and output shape

Prefill cost scales with prompt length and decode cost with generation length, so a benchmark that lets both float is measuring its own traffic mix. Pin them. I would run a small grid rather than one point, because the shape of the curve is the finding.

A warm engine

The first requests pay for CUDA graph capture, autotuning and cache population. Discard them explicitly rather than hoping the average absorbs them, and say how many you discarded.

The whole distribution

Report p50, p95 and p99 for TTFT and inter-token latency, not a mean. The mean hides exactly the behaviour that makes a deployment feel bad, and the gap between p50 and p99 is usually the interesting result.

The run

reproducible or it did not happen

Server first, pinned to one placement so the result means something. This is the GLM-5.3-Flash configuration from the frontier pages, at the tensor parallel degree the fleet arithmetic recommends.

# one replica, half a node, no TP traffic across a NIC
vllm serve zai-org/GLM-5.3-Flash \
  --tensor-parallel-size 4 \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --gpu-memory-utilization 0.92 \
  --max-model-len 131072 \
  --disable-log-requests

Then sweep concurrency against a fixed shape. The point is the curve, not any single row of it.

for C in 1 4 16 64 128 256; do
  vllm bench serve \
    --model zai-org/GLM-5.3-Flash \
    --dataset-name random \
    --random-input-len 4096 \
    --random-output-len 512 \
    --max-concurrency "$C" \
    --num-prompts $(( C * 12 )) \
    --ignore-eos \
    --percentile-metrics ttft,tpot,itl,e2el \
    --metric-percentiles 50,95,99 \
    --save-result --result-filename "glm53f_tp4_c$C.json"
done

--ignore-eos matters. Without it the model stops when it wants to and your output lengths vary, which means you are no longer measuring a fixed shape and the throughput number drifts with the prompt.
Scale prompts with concurrency. A fixed prompt count at high concurrency finishes before the engine reaches steady state, and you end up reporting the ramp.
Record the environment. Engine version, driver, container digest, and the exact model revision. I have watched a 15% swing turn out to be a point release, and without the version pinned there was no way to prove it.

Ways I have seen this go wrong

including my own

Benchmarking the client

A single-threaded load generator saturates before the server does, and then you are measuring Python. If throughput plateaus while GPU utilisation sits low, suspect the harness before the engine. Run the client on a separate host and check it can generate more load than the server can absorb.

Counting tokens two different ways

Server-side output token counts and client-side counts disagree when the tokenizer differs or when the client counts SSE frames rather than tokens. Comparing two engines on numbers gathered differently is the most common way to produce a confident, wrong conclusion.

Forgetting where the clock starts

TTFT measured at the client includes network, load balancer and tokenization. Measured at the server it does not. Both are legitimate and they are not the same number, so say which one you are reporting, especially when comparing against a hosted API where you only ever get the first kind.

Reporting one run

Placement, thermals and neighbours all move the result. I would run each point three times and report the spread, because a single run that happens to land on a quiet node is not a measurement, it is a draw from a lottery I did not know I had entered.

What I would publish

the empty table

When the run happens, this is where it goes. Deliberately empty rather than filled with a plausible guess, because the entire point of the exercise is that these are the only numbers on the site that would be mine.

ConcurrencyTTFT p50 / p99ITL p50 / p99Output tok/sTok/s/user

1pendingpendingpendingpending

16pendingpendingpendingpending

64pendingpendingpendingpending

256pendingpendingpendingpending

GLM-5.3-Flash, TP=4, FP8 weights and FP8 KV cache, 4,096 in and 512 out, on one HGX B200 node. Nothing here is estimated: every cell stays empty until it is measured.