On November 30, 2022, there were maybe a few hundred people in the world who did what I do now. They worked at OpenAI, Midjourney, Anthropic, or the big labs, and it looked like that was just how the industry would work: a handful of companies train models, everyone else rents intelligence a token at a time.
Three years later, that's not what happened. Training models is hard and expensive, but not so hard or expensive that only a handful of players can do it. There are now more than two million open models on Hugging Face, 25 times more than five years ago. Every engineer can deploy their own intelligence.
And that changed the job. When you can deploy your own models, inference stops being a black box you pay for and starts being a discipline you control. That's what I do: make models faster, cheaper, and more reliable in production.
The three layers
Inference engineering lives in three layers, and this whole journey follows them:
- Runtime - the single GPU. Kernels, attention, KV cache, quantization, engines. This is where most of the leverage lives.
- Infrastructure - many GPUs. Clusters, autoscaling, routing, multi-cloud, reliability.
- Tooling - the layer that makes the other two debuggable. Benchmarking, observability, client code.
Most people start in the middle, at the engine level, and never go down to the kernel or up to the cluster. That's fine for shipping a demo. It's not enough for shipping something that survives a traffic spike.
The rules I'm setting for myself
- Build, don't copy. Every concept gets built from scratch or measured directly before I write about it.
- One idea per day. No 5,000-word sprawl. A single mental model, explained properly, with a visualization.
- Honest about tradeoffs. Every technique has a "when it fails" section, because they all have one.
- Write it like I'd say it. This is a field notebook, not a textbook. If it reads like a textbook, I've failed.
Doing inference well requires three layers: Runtime, Infrastructure, and Tooling.
Philip Kiely, Inference Engineering
Tomorrow I start at the bottom: what actually happens inside a model when you ask it a question. No magic, just matrix multiplication and a very patient attention mechanism.
See you on the next post.