Why write an inference engine in Rust? Because Rust gives you C++ performance with memory safety, and for a serving engine, that's the whole game. No garbage collector, no interpreter, no runtime surprises. The engine compiles to a single binary that loads a model and serves tokens.
This is the story of building one, and running it on B200 in disaggregated setups.
Why Rust for inference
- Performance. Rust compiles to native code with no runtime overhead. The hot path (the decode loop) is as fast as C++.
- Memory safety. The borrow checker catches use-after-free and data races at compile time. In a serving engine that runs for months, that matters.
- Concurrency. Rust's async model (tokio) handles thousands of concurrent requests cleanly, which is exactly what a serving engine needs.
- Single binary. Deploy one artifact, no Python, no dependency tree. This is the CVE story from the compiled-kernels note, taken further.
The architecture
The components
- tokio server. Async HTTP with SSE streaming. Handles thousands of concurrent connections.
- Scheduler. Continuous batching, the same idea as vLLM, implemented in Rust.
- CUDA kernels. The attention and GEMM kernels, written in CUDA C and called from Rust via FFI. The hot path is native.
- KV cache manager. Paged, prefix-aware, the Mooncake-style cache pool integration.
The Rust + CUDA pattern
The canonical architecture for high-performance Rust inference is Rust orchestration over raw CUDA kernels, via the cudarc crate (safe Rust bindings for the entire CUDA toolkit: cuBLASLt, NCCL, cuDNN, NVRTC, cuFILE/GDS, CUPTI). Candle's kernel library demonstrates the exact pattern: hand-written .cu files for matmul, attention, quantized GEMM, and MoE dispatch, wrapped in safe Rust.
Key pieces for B200 disaggregated serving: NCCL bindings for inter-GPU KV-cache transfer, cuBLASLt for Blackwell's FP4/FP8 tensor core GEMMs, cuFILE/GDS for zero-copy KV-cache persistence, and dynamic loading so the same binary runs across CUDA 12.x and 13.x.
What running on B200 means
B200's FP4 and dual-die architecture are the reason a Rust engine makes sense. The engine compiles to native code that drives the tensor cores directly, no framework in between. In disaggregated setups, the engine runs on decode nodes, streaming tokens while the KV cache pool holds the context.
The proof it works: mistral.rs (a production Rust engine) benchmarks on B200 show 27,706 TPS prefill (Q8) vs llama.cpp's 11,992, and 43,548 TPS BF16 prefill vs vLLM's 39,431. A Rust engine can match or beat the established C++/Python engines on Blackwell. The result: frontier models served from a single binary, with the latency and throughput of a hand-tuned C++ engine and the safety of Rust.
Sources
- mistral.rs: production Rust engine with B200 benchmarks.
- candle: HuggingFace's Rust ML framework with hand-written CUDA kernels.
- cudarc: safe Rust bindings for the entire CUDA toolkit.
- burn: backend-agnostic Rust DL framework.
- DistServe: the disaggregation blueprint.
- Mooncake: the KV-cache-centric architecture.
- Splitwise: the phase-splitting analysis.
- monoio: io_uring-based runtime for ultra-low-latency networking.