Build it: FastAPI + vLLM with health checks

Yesterday in the design post, I laid out the full inference stack architecture. Today, we build the core piece: the inference server itself. This is a FastAPI application that wraps vLLM's async engine, exposes an OpenAI-compatible API, implements proper health checks for Kubernetes, streams tokens via SSE, and exports Prometheus metrics.

This is real code. I have run variations of this in production. Let us build it layer by layer.

Project structure

inference-server/
├── app/
│   ├── __init__.py
│   ├── main.py          # FastAPI app, lifespan, routes
│   ├── engine.py         # vLLM engine wrapper
│   ├── health.py         # Health check logic
│   ├── metrics.py        # Prometheus metrics
│   └── config.py         # Configuration from env vars
├── Dockerfile
├── requirements.txt
└── docker-compose.yaml

Configuration

Everything configurable comes from environment variables. No magic config files that get out of sync:

# app/config.py
from pydantic_settings import BaseSettings

class Settings(BaseSettings):
    model_name: str = "meta-llama/Llama-3.1-8B-Instruct"
    tensor_parallel_size: int = 1
    max_model_len: int = 8192
    gpu_memory_utilization: float = 0.90
    dtype: str = "auto"  # auto selects bf16 on Ampere+
    quantization: str | None = None  # "awq", "gptq", "fp8"
    host: str = "0.0.0.0"
    port: int = 8000
    max_concurrent_requests: int = 256

    class Config:
        env_prefix = "INFERENCE_"

settings = Settings()

The gpu_memory_utilization of 0.90 leaves 10% of GPU memory as headroom. I have seen OOM crashes at 0.95 when KV cache demand spikes. Better to leave room and let the scheduler reject requests gracefully than to crash the process.

The vLLM engine wrapper

vLLM's AsyncLLMEngine is the core. We wrap it to manage its lifecycle and expose a clean interface:

# app/engine.py
from vllm.engine.async_llm_engine import AsyncLLMEngine
from vllm.engine.arg_utils import AsyncEngineArgs
from vllm.sampling_params import SamplingParams
from app.config import settings

_engine: AsyncLLMEngine | None = None

async def initialize_engine():
    global _engine
    engine_args = AsyncEngineArgs(
        model=settings.model_name,
        tensor_parallel_size=settings.tensor_parallel_size,
        max_model_len=settings.max_model_len,
        gpu_memory_utilization=settings.gpu_memory_utilization,
        dtype=settings.dtype,
        quantization=settings.quantization,
        enable_chunked_prefill=True,
        enable_prefix_caching=True,
    )
    _engine = AsyncLLMEngine.from_engine_args(engine_args)

def get_engine() -> AsyncLLMEngine:
    if _engine is None:
        raise RuntimeError("Engine not initialized")
    return _engine

async def shutdown_engine():
    global _engine
    _engine = None

Two flags worth noting: enable_chunked_prefill=True activates the chunked prefill scheduling we discussed, and enable_prefix_caching=True turns on prefix caching for KV reuse across requests with shared prompts.

Health checks: liveness vs readiness

This is the piece most tutorials skip and most production incidents stem from. Kubernetes (and any load balancer) needs two different health signals:

• Liveness: "Is the process alive and not deadlocked?" If this fails, Kubernetes restarts the pod. It should be cheap and never depend on the model.
• Readiness: "Is the model loaded and ready to serve traffic?" If this fails, the pod is removed from the service's endpoint list but not restarted. This is critical during startup: a vLLM server can take 30 to 120 seconds to load a large model, and you do not want traffic hitting it during that window.

# app/health.py
from app.engine import get_engine
import time

_startup_complete = False
_last_healthy_time = time.monotonic()

async def mark_ready():
    global _startup_complete
    _startup_complete = True

async def liveness_check() -> dict:
    """Cheap check: is the process responsive?"""
    return {"status": "alive", "uptime_s": time.monotonic()}

async def readiness_check() -> dict:
    """Is the model loaded and the engine accepting requests?"""
    if not _startup_complete:
        raise RuntimeError("Model still loading")

    engine = get_engine()
    # Check that the engine's background loop is running
    if not engine.is_running:
        raise RuntimeError("Engine loop stopped")

    # Check GPU memory pressure
    # vLLM exposes this through engine internals
    global _last_healthy_time
    _last_healthy_time = time.monotonic()

    return {
        "status": "ready",
        "model": engine.engine.model_config.model,
    }

The FastAPI application

Now the main app, tying everything together with a lifespan context manager for clean startup and shutdown:

# app/main.py
import uuid
import time
import asyncio
from contextlib import asynccontextmanager
from fastapi import FastAPI, HTTPException, Request
from fastapi.responses import StreamingResponse, JSONResponse
from pydantic import BaseModel
from vllm.sampling_params import SamplingParams

from app.engine import initialize_engine, get_engine, shutdown_engine
from app.health import liveness_check, readiness_check, mark_ready
from app.metrics import (
    REQUEST_COUNT, REQUEST_LATENCY, TOKENS_GENERATED,
    ACTIVE_REQUESTS, start_metrics_server
)
from app.config import settings

@asynccontextmanager
async def lifespan(app: FastAPI):
    # Startup: load model (this takes 30-120s for large models)
    print(f"Loading model: {settings.model_name}")
    await initialize_engine()
    await mark_ready()
    print("Model loaded, server ready")
    start_metrics_server()
    yield
    # Shutdown: clean up
    await shutdown_engine()
    print("Engine shut down")

app = FastAPI(title="Inference Server", lifespan=lifespan)

# ---------- Health endpoints ----------

@app.get("/health/live")
async def health_live():
    return await liveness_check()

@app.get("/health/ready")
async def health_ready():
    try:
        return await readiness_check()
    except RuntimeError as e:
        raise HTTPException(status_code=503, detail=str(e))

# ---------- Chat completions (OpenAI-compatible) ----------

class Message(BaseModel):
    role: str
    content: str

class ChatRequest(BaseModel):
    model: str = "default"
    messages: list[Message]
    temperature: float = 0.7
    top_p: float = 1.0
    max_tokens: int = 512
    stream: bool = False

@app.post("/v1/chat/completions")
async def chat_completions(request: ChatRequest):
    REQUEST_COUNT.labels(model=request.model).inc()
    ACTIVE_REQUESTS.labels(model=request.model).inc()
    start_time = time.monotonic()

    try:
        engine = get_engine()
        request_id = str(uuid.uuid4())

        # Build prompt from messages
        prompt = format_chat_prompt(request.messages)

        sampling_params = SamplingParams(
            temperature=request.temperature,
            top_p=request.top_p,
            max_tokens=request.max_tokens,
        )

        if request.stream:
            return StreamingResponse(
                stream_response(engine, prompt, sampling_params,
                                request_id, request.model, start_time),
                media_type="text/event-stream",
            )
        else:
            # Non-streaming: collect all tokens
            full_output = ""
            token_count = 0
            async for output in engine.generate(prompt, sampling_params,
                                                 request_id):
                if output.outputs:
                    full_output = output.outputs[0].text
                    token_count = len(output.outputs[0].token_ids)

            elapsed = time.monotonic() - start_time
            REQUEST_LATENCY.labels(model=request.model).observe(elapsed)
            TOKENS_GENERATED.labels(model=request.model).inc(token_count)

            return {
                "id": request_id,
                "object": "chat.completion",
                "choices": [{
                    "index": 0,
                    "message": {"role": "assistant", "content": full_output},
                    "finish_reason": "stop"
                }],
                "usage": {
                    "prompt_tokens": len(prompt.split()),  # approximate
                    "completion_tokens": token_count,
                }
            }
    finally:
        ACTIVE_REQUESTS.labels(model=request.model).dec()

Streaming with SSE

The streaming generator yields server-sent events in the OpenAI format:

import json

async def stream_response(engine, prompt, sampling_params,
                           request_id, model, start_time):
    previous_text = ""
    token_count = 0

    async for output in engine.generate(prompt, sampling_params,
                                         request_id):
        if output.outputs:
            current_text = output.outputs[0].text
            delta = current_text[len(previous_text):]
            previous_text = current_text
            token_count = len(output.outputs[0].token_ids)

            if delta:
                chunk = {
                    "id": request_id,
                    "object": "chat.completion.chunk",
                    "choices": [{
                        "index": 0,
                        "delta": {"content": delta},
                        "finish_reason": None
                    }]
                }
                yield f"data: {json.dumps(chunk)}\n\n"

    # Final chunk with finish_reason
    final_chunk = {
        "id": request_id,
        "object": "chat.completion.chunk",
        "choices": [{
            "index": 0,
            "delta": {},
            "finish_reason": "stop"
        }]
    }
    yield f"data: {json.dumps(final_chunk)}\n\n"
    yield "data: [DONE]\n\n"

    elapsed = time.monotonic() - start_time
    REQUEST_LATENCY.labels(model=model).observe(elapsed)
    TOKENS_GENERATED.labels(model=model).inc(token_count)

Prometheus metrics

You cannot operate what you cannot measure. Four metrics cover most production needs:

# app/metrics.py
from prometheus_client import Counter, Histogram, Gauge, start_http_server

REQUEST_COUNT = Counter(
    "inference_requests_total",
    "Total inference requests",
    ["model"]
)

REQUEST_LATENCY = Histogram(
    "inference_request_duration_seconds",
    "Request latency in seconds",
    ["model"],
    buckets=[0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, 30.0]
)

TOKENS_GENERATED = Counter(
    "inference_tokens_generated_total",
    "Total tokens generated",
    ["model"]
)

ACTIVE_REQUESTS = Gauge(
    "inference_active_requests",
    "Currently in-flight requests",
    ["model"]
)

def start_metrics_server(port: int = 9090):
    start_http_server(port)

The metrics server runs on port 9090, separate from the API on 8000. Prometheus scrapes 9090; your load balancer hits 8000. Keep them separate so metrics collection never competes with serving.

The Dockerfile

The container image builds on NVIDIA's PyTorch base (which includes CUDA, cuDNN, and NCCL) and installs vLLM:

FROM nvcr.io/nvidia/pytorch:24.07-py3

WORKDIR /app

# Install dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy application code
COPY app/ app/

# Health check for Docker (not Kubernetes)
HEALTHCHECK --interval=30s --timeout=10s --retries=3 \
    CMD curl -f http://localhost:8000/health/ready || exit 1

# Metrics on 9090, API on 8000
EXPOSE 8000 9090

# Use uvicorn with multiple workers is NOT recommended for vLLM
# vLLM manages its own async engine; use a single worker
CMD ["uvicorn", "app.main:app", \
     "--host", "0.0.0.0", \
     "--port", "8000", \
     "--workers", "1", \
     "--timeout-keep-alive", "120"]

Kubernetes manifests

For completeness, here are the key pieces of the Kubernetes deployment:

# Key sections of the deployment spec
spec:
  containers:
  - name: inference
    image: inference-server:latest
    resources:
      limits:
        nvidia.com/gpu: 1
    env:
    - name: INFERENCE_MODEL_NAME
      value: "meta-llama/Llama-3.1-8B-Instruct"
    - name: INFERENCE_GPU_MEMORY_UTILIZATION
      value: "0.90"
    ports:
    - containerPort: 8000
      name: api
    - containerPort: 9090
      name: metrics
    livenessProbe:
      httpGet:
        path: /health/live
        port: 8000
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 5
    readinessProbe:
      httpGet:
        path: /health/ready
        port: 8000
      initialDelaySeconds: 60      # model loading takes time
      periodSeconds: 10
      timeoutSeconds: 5
      failureThreshold: 30         # allow up to 5 min for loading
    startupProbe:
      httpGet:
        path: /health/ready
        port: 8000
      initialDelaySeconds: 30
      periodSeconds: 10
      failureThreshold: 60         # 10 min max startup

The startup probe is the hero here. It gives the model up to 10 minutes to load before Kubernetes considers the pod failed. Without it, the readiness probe starts immediately and the pod gets restarted before the model finishes loading.

Testing the deployment

Once the container is running, verify it end to end:

# Check health
curl http://localhost:8000/health/ready
# {"status": "ready", "model": "meta-llama/Llama-3.1-8B-Instruct"}

# Non-streaming request
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "What is KV cache?"}],
    "max_tokens": 128
  }'

# Streaming request
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -N \
  -d '{
    "messages": [{"role": "user", "content": "Explain PagedAttention"}],
    "max_tokens": 256,
    "stream": true
  }'

# Check metrics
curl http://localhost:9090/metrics | grep inference_

Then run the eval harness against the endpoint to verify quality. Then run a load test with Locust to find the saturation point. Then set up blue-green deployment so you can roll out model updates without downtime.

This is where 99 days of learning converge: a container you can deploy, scale, monitor, and trust. The code is straightforward because we understand every layer beneath it. That understanding is the real deliverable of this series.

That is the inference stack, built from the ground up. Every decision, from the roofline model to chunked prefill to this FastAPI wrapper, traces back to first principles. Build it, measure it, improve it.