Containers and NIMs

The first time I tried to deploy a model on a new machine, I spent four hours debugging CUDA version mismatches. The model worked perfectly on my development box. On the production server, the CUDA runtime was 12.1, the driver was 535, the PyTorch build expected 11.8, and nothing would load. That was the day I became a container believer.

Why containers for inference

LLM inference has a uniquely painful dependency stack:

• NVIDIA driver: must be compatible with the CUDA runtime version
• CUDA toolkit: must match the compiled PyTorch/TensorRT version
• cuDNN, cuBLAS, NCCL: each with its own version matrix
• Python environment: PyTorch, transformers, vLLM or TensorRT-LLM, and dozens of transitive dependencies
• Model weights: often tens of gigabytes, sometimes hundreds

A Docker container freezes all of this (except the host driver) into a single reproducible image. The NVIDIA Container Toolkit (nvidia-docker) exposes the host GPU to the container through a runtime hook, so the container sees the GPU as if it were running on bare metal.

# Run a vLLM container with GPU access
docker run --gpus all \
  -v /models/llama-3-8b:/model \
  -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model /model \
  --tensor-parallel-size 1 \
  --dtype float16 \
  --max-model-len 4096

The --gpus all flag tells the NVIDIA runtime to expose all GPUs to the container. You can also specify individual GPUs with --gpus '"device=0,1"' for multi-GPU serving or isolation.

The container image problem

Inference containers are large. A minimal vLLM image with CUDA and PyTorch is 8 to 12 GB. Add model weights (say, 16 GB for Llama 3 8B in FP16) and you are at 24+ GB. For a 70B model, the weights alone are 140 GB. This creates real operational problems:

• Pull times: downloading a 20 GB container image over a 1 Gbps network takes over 2.5 minutes. On cold starts, this delay is the dominant contributor to startup latency.
• Storage: each node needs enough local storage for the image layers plus model weights. For multi-model deployments, storage costs add up.
• Layer caching: Docker's layer cache helps when you update the serving code without changing the base image. Structure your Dockerfile so that CUDA, PyTorch, and dependencies are in lower layers (which change rarely) and your serving code is in upper layers (which change often).

# Dockerfile structure for inference
# Layer 1: CUDA base (rarely changes)
FROM nvcr.io/nvidia/cuda:12.4.0-runtime-ubuntu22.04

# Layer 2: Python + PyTorch (changes with framework updates)
RUN pip install torch==2.3.0 --index-url https://download.pytorch.org/whl/cu124

# Layer 3: Inference framework (changes more often)
RUN pip install vllm==0.5.0

# Layer 4: Your serving code (changes frequently)
COPY serve.py /app/serve.py

# Model weights mounted at runtime, not baked into image
# docker run -v /models:/models ...

The key practice: never bake model weights into the container image. Mount them as volumes or download them at startup. This keeps the image size manageable and lets you swap models without rebuilding.

NVIDIA NIMs: pre-optimized containers

NVIDIA Inference Microservices (NIMs) are pre-built, pre-optimized container images that package a model with its ideal serving runtime and configuration. Instead of building your own container, choosing the right quantization, tuning batch sizes, and configuring the runtime, you pull a NIM and run it.

A NIM container includes:

• The model weights (or downloads them on first run from NGC)
• A pre-compiled TensorRT-LLM engine optimized for the target GPU
• The serving runtime (Triton Inference Server) with tuned configuration
• An OpenAI-compatible API endpoint
• Health checks, metrics, and readiness probes

# Run a Llama 3 8B NIM
docker run --gpus all \
  -e NGC_API_KEY=$NGC_API_KEY \
  -p 8000:8000 \
  nvcr.io/nim/meta/llama-3.1-8b-instruct:latest

# It exposes an OpenAI-compatible endpoint
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta/llama-3.1-8b-instruct",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

The value proposition is clear: NVIDIA's engineers have already done the optimization work. The TensorRT-LLM engine inside the NIM is compiled for specific GPU profiles (H100 SXM, H100 PCIe, A100, etc.) with the best quantization settings, batch configurations, and kernel selections. For teams that want production inference without becoming inference experts, NIMs significantly reduce time-to-deployment.

NIMs vs. building your own

The trade-off is control vs. convenience:

• NIMs: fastest path to production. Best throughput out of the box. Limited customization (you cannot easily change the quantization scheme, attention implementation, or batching strategy). Requires an NVIDIA AI Enterprise license for production use.
• Custom containers (vLLM, SGLang, TensorRT-LLM): full control over every parameter. More work to optimize. No licensing constraints for open-source options. Better for teams with specific requirements or novel architectures.

In my experience, NIMs are excellent for standard models (Llama, Mistral, Mixtral) where NVIDIA has invested in optimization. For custom fine-tuned models, novel architectures, or workloads with unusual requirements (very long context, specific structured output constraints), building your own container with vLLM or SGLang gives you the flexibility you need.

Kubernetes and inference containers

In Kubernetes, inference containers run as Deployments or StatefulSets with GPU resource requests. The NVIDIA GPU Operator automates driver installation, device plugin deployment, and container runtime configuration across the cluster.

Key considerations for Kubernetes-based inference:

• Resource requests: always set nvidia.com/gpu limits. Without them, the scheduler cannot properly place GPU workloads.
• Readiness probes: model loading can take minutes. Set a generous initialDelaySeconds on your readiness probe to avoid the kubelet killing the pod during model load.
• Shared memory: many inference frameworks use shared memory (/dev/shm) for inter-process communication. Set shm-size or mount a Memory-backed emptyDir at /dev/shm.

Containers solve "works on my machine." NIMs solve "I do not want to become an inference optimization expert." Both are valid, and knowing when to use each is an infrastructure skill in itself.

Next: autoscaling, where we tackle the question of how many containers to run, when to scale up, and how to handle cold starts without blowing your budget.