It's done, and it's real: a full production inference stack, built from scratch over the course of this journey, deployed, measured, and documented. Not a demo. A system that serves traffic, survives restarts, and reports its own health.
This post is the reflection. The build was the point, but the reflection is where the learning lives.
What I built
I assembled the capstone: a FastAPI service in front of a vLLM engine, with health checks, a metrics endpoint, and a Grafana dashboard. The pieces:
- vLLM as the engine, configured with the lessons from the runtime phase: gpu_memory_utilization, max_num_seqs, prefix caching on.
- FastAPI as the control plane, handling auth, routing, and the streaming response.
- Prometheus metrics from the engine: TTFT, TBT, queue depth, KV cache hit rate.
- Grafana to see it all, the dashboard from the observability post.
- Autoscaling tuned on the queue-depth principle.
The whole thing runs on one GPU, serves a realistic workload, and the numbers are in the charts, not in my head.
What I learned, ranked
- The roofline is the master key. Every single optimization I did traces to it. If I'd internalized it at the start, I'd have saved myself weeks.
- Memory management is the real discipline. KV cache, PagedAttention, prefix caching, quantization. Inference is a memory problem wearing a compute costume.
- Measure before you optimize. My first instinct was always wrong. The profiler was right. Every time.
- Batching is the cheapest win. Continuous batching alone took utilization from 50% to 85% in my tests. No new hardware, no new math.
- Cost is a first-class metric. Dollars per token, not just latency and throughput. The best system is the one that meets the SLA for the least money.
What I'd do differently
- Start with the roofline. I put it early. It should have been first.
- Build the benchmark harness earlier. It was too late. A reusable harness from the start would have made every comparison honest.
- Spend less time on engine wars. vLLM vs SGLang vs TensorRT-LLM is a fun debate and a bad use of time. They converge. The principles don't.
Where the field is going
Three directions I'm watching, all of which showed up in the last ten days:
- Disaggregation everywhere. Prefill and decode on separate machines, because they have opposite roofline profiles. Dynamo is making this standard.
- Speculation as default. EAGLE and Medusa are becoming default engine features, not exotic tricks.
- Modality convergence. One engine serving text, image, video, and speech, with a shared scheduler. The cost curves are getting interesting.
The honest close
I started this journey thinking inference was about kernels and FLOPS. It's actually about throughput per dollar and latency per user, and the kernels are just the means. Now I can say the field is young, the problems are hard, and the people solving them are the most underrated engineers in the industry.
If you made it this far, thanks for reading. Now go build something and measure it.