I build, measure, and operate inference.
Inference Engineer at Sarvam. I live on the stack between CUDA and Kubernetes, and I write down everything I learn so the next person doesn't have to learn it the hard way.
What I do at Sarvam
My day-to-day is a mix of model serving and infrastructure. I run the serving stack for production models: vLLM, SGLang, TensorRT-LLM, and Triton, on GPU clusters that span clouds. When a model is too slow, too expensive, or too flaky, I'm the one who finds out why.
That usually means profiling kernels, tuning quantization, chasing KV-cache bottlenecks, and debugging the autoscaling policy that decided to spin down a replica right before a traffic spike.
A day in the life
- Morning triage. Check dashboards, alerts, and queue depth. Is anything slow, and is it the model or the infrastructure?
- Deep work. Profiling a kernel, tuning a quantization config, or benchmarking a new engine against our traffic.
- Build. Containerizing a new model, wiring autoscaling, adding observability so the next incident is shorter.
- Operate. Deploy, watch the canary, measure TTFT and TPS against the SLA, iterate.
- Write. Every hard-won lesson becomes a post. If I can't explain it to a reader, I don't understand it well enough.
The hard problems I'm chasing
- Disaggregation that actually pays. Splitting prefill and decode is easy on paper; making the KV cache handoff fast enough that users never notice is the hard part. The tradeoff between cache locality and GPU utilization is a moving target, and the answer changes with every model release.
- Prefix caching at fleet scale. When a thousand agents share a system prompt, cache hits are free money. But the cache pool has to be shared across machines, evicted intelligently, and sized against memory that could otherwise serve more requests. The hit-rate-to-latency curve is where the real engineering lives.
- Speculative decoding that survives production. The paper numbers are beautiful; the reality of draft-model latency variance, KV-cache pressure, and batch interference is not. When does the acceptance rate justify the extra memory, and how do you keep it from regressing under load?
- Cost per token, honestly. GPUs are expensive, and the difference between a good and a great deployment is often 2x on the same hardware. But utilization, power, and amortized fleet cost are all moving pieces. The real question is not "what does one token cost" but "what does a reliable token cost at the 99th percentile under bursty traffic."
- Observability that explains, not just displays. You can't fix what you can't see, but dashboards full of numbers you don't act on are just expensive wallpaper. The question is always "why is it slow", and the answer is rarely one number. Correlating TTFT spikes with scheduler decisions, cache evictions, and network jitter is where the insight lives.
- Autoscaling under bursty, correlated load. Agents and batch jobs arrive in waves. The autoscaler has to predict, not react. Cold starts on a 141GB model are measured in minutes, not seconds, so the policy has to be right before the spike, not after.