Production Dockerfile for vLLM

The simplest way to run vLLM is pip install vllm && vllm serve. That works for benchmarking and development, but production needs more: reproducible builds, health checks, security hardening, and a container that starts quickly and fails gracefully. Today I will walk through the Dockerfile I use for production vLLM deployments.

The base image choice

vLLM publishes official Docker images, but I prefer building my own for two reasons: I control the exact versions of CUDA, PyTorch, and vLLM, and I can bake the model weights into the image for faster cold starts.

The base image matters more than you might think. You need CUDA runtime libraries, cuBLAS, cuDNN, and NCCL for multi-GPU. NVIDIA's official CUDA images come in three flavors: base (runtime only), runtime (runtime + cuBLAS), and devel (everything + compilers). For serving, runtime is sufficient and saves several GB over devel.

# Stage 1: Install vLLM and dependencies
FROM nvidia/cuda:12.4.1-runtime-ubuntu22.04 AS builder

ENV DEBIAN_FRONTEND=noninteractive
ENV PYTHONDONTWRITEBYTECODE=1

RUN apt-get update && apt-get install -y --no-install-recommends \
    python3.11 python3.11-venv python3-pip \
    && rm -rf /var/lib/apt/lists/*

RUN python3.11 -m venv /opt/vllm-env
ENV PATH="/opt/vllm-env/bin:$PATH"

# Pin exact versions for reproducibility
RUN pip install --no-cache-dir \
    vllm==0.6.4 \
    torch==2.5.1 \
    triton==3.1.0

Baking model weights into the image

There are two approaches to model weights: download them at container start, or bake them into the image. Downloading at start is flexible (change the model without rebuilding) but adds 30 to 120 seconds to cold start time for a 7B to 70B model. Baking them in makes the image large but the container starts immediately.

For production, I bake the weights in. Cold start time matters for autoscaling, and a 20 GB image that starts in 10 seconds is better than a 2 GB image that downloads weights for 60 seconds.

# Stage 2: Download model weights
FROM builder AS model-downloader

RUN pip install --no-cache-dir huggingface_hub

ARG MODEL_ID=meta-llama/Llama-3.1-8B-Instruct
ARG HF_TOKEN

# Download model weights into the image
RUN --mount=type=secret,id=hf_token \
    HF_TOKEN=$(cat /run/secrets/hf_token) \
    huggingface-cli download ${MODEL_ID} \
    --local-dir /models/${MODEL_ID} \
    --local-dir-use-symlinks False

The production runtime stage

The final stage assembles everything with security and operational concerns:

# Stage 3: Production runtime
FROM nvidia/cuda:12.4.1-runtime-ubuntu22.04

ENV DEBIAN_FRONTEND=noninteractive
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1

RUN apt-get update && apt-get install -y --no-install-recommends \
    python3.11 python3.11-venv curl \
    && rm -rf /var/lib/apt/lists/*

# Copy venv and model from builder stages
COPY --from=builder /opt/vllm-env /opt/vllm-env
COPY --from=model-downloader /models /models

ENV PATH="/opt/vllm-env/bin:$PATH"

# Non-root user for security
RUN groupadd -r vllm && useradd -r -g vllm -d /home/vllm -s /bin/bash vllm
RUN mkdir -p /home/vllm && chown -R vllm:vllm /home/vllm
USER vllm

# Default model path (overridable)
ENV MODEL_PATH=/models/meta-llama/Llama-3.1-8B-Instruct

# Health check: vLLM exposes /health on the API server
HEALTHCHECK --interval=30s --timeout=10s --start-period=120s --retries=3 \
    CMD curl -f http://localhost:8000/health || exit 1

EXPOSE 8000

ENTRYPOINT ["python3.11", "-m", "vllm.entrypoints.openai.api_server"]
CMD ["--model", "/models/meta-llama/Llama-3.1-8B-Instruct", \
     "--host", "0.0.0.0", \
     "--port", "8000", \
     "--max-model-len", "8192", \
     "--gpu-memory-utilization", "0.9", \
     "--enable-prefix-caching"]

Key production details

Let me call out the decisions that are easy to overlook:

HEALTHCHECK with start-period. vLLM takes time to load the model and compile CUDA graphs. The --start-period=120s gives it two minutes before the orchestrator starts checking health. Without this, Kubernetes will kill the pod before it finishes loading. The /health endpoint returns 200 only after the model is loaded and ready to serve.

Non-root user. Running as root inside a container is a security anti-pattern. The vllm user has no special privileges, which limits the blast radius if the container is compromised. The model files are owned by root and read-only to the vllm user, which is intentional.

PYTHONUNBUFFERED=1. Without this, Python buffers stdout and you do not see logs until the buffer flushes. In a container orchestrator, this means you might not see error messages when the container crashes.

gpu-memory-utilization=0.9. By default, vLLM tries to use 90% of GPU memory for the KV cache. This is the right setting for dedicated serving, but if you are sharing the GPU with monitoring or other processes, lower it to 0.85.

enable-prefix-caching. This is essentially free performance for workloads with shared system prompts, which is nearly every production deployment. It uses the hash-based prefix dedup I covered earlier.

Building and testing

# Build with Docker BuildKit for secret mounting
DOCKER_BUILDKIT=1 docker build \
    --secret id=hf_token,src=$HOME/.cache/huggingface/token \
    --build-arg MODEL_ID=meta-llama/Llama-3.1-8B-Instruct \
    -t vllm-prod:latest .

# Test locally (requires GPU)
docker run --gpus all -p 8000:8000 vllm-prod:latest

# Verify health
curl http://localhost:8000/health
# {"status": "ok"}

# Test inference
curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{"model": "meta-llama/Llama-3.1-8B-Instruct",
         "prompt": "The capital of France is",
         "max_tokens": 32}'

What I would add for Kubernetes

When deploying to Kubernetes, I add a few things on the orchestration side rather than in the Dockerfile:

• Readiness probe: Hit /health and only route traffic after the model is loaded. This is separate from the liveness probe, which uses the same endpoint but with different failure thresholds.
• Resource limits: Set nvidia.com/gpu: 1 in the resource requests to ensure the pod gets a GPU. Set memory limits high enough for the model weights plus KV cache plus overhead.
• Graceful shutdown: vLLM handles SIGTERM by finishing in-flight requests. Set terminationGracePeriodSeconds to at least 60 seconds to allow long generations to complete.

Tomorrow: building a NIM-compatible container, which takes this Dockerfile pattern and adapts it to NVIDIA's container standard for enterprise deployment.