Measure cold start latency

Cold start latency determines how quickly your autoscaler can respond to a traffic spike. If your LLM container takes 3 minutes to start, your autoscaler needs to anticipate load 3 minutes in advance. If it takes 30 seconds, the system is far more responsive. Understanding where that time goes is the first step to reducing it.

Anatomy of a cold start

When a vLLM container launches, it goes through several distinct phases before it can serve its first request. I instrumented each phase to measure where the time goes:

1. Container initialization: The container runtime sets up the filesystem, mounts volumes, and starts the process. This is typically 1 to 3 seconds and there is little you can do about it.
2. Python startup: The Python interpreter loads, imports are resolved, and the vLLM module initializes. This takes 3 to 8 seconds, mostly spent importing PyTorch and its CUDA bindings.
3. CUDA context creation: The first CUDA call initializes the GPU context. This involves loading the CUDA runtime, initializing the driver, and allocating GPU resources. On an H100, this takes 2 to 5 seconds.
4. Model weight loading: This is usually the dominant phase. Weights are read from disk (or downloaded from a model store) and transferred to GPU memory.
5. KV cache allocation: vLLM pre-allocates the KV cache blocks based on available GPU memory. This is fast (under 1 second) but requires the model to be loaded first so it knows how much memory is left.
6. CUDA graph capture: If CUDA graphs are enabled (they are by default in vLLM), the engine captures execution graphs for common batch sizes. This adds 5 to 15 seconds but eliminates kernel launch overhead during serving.

Measuring with a script

I wrote a simple measurement script that starts a vLLM server and polls the health endpoint until it responds:

#!/bin/bash
# measure_cold_start.sh - Time from container start to first healthy response

IMAGE="vllm-prod:latest"
PORT=8000

echo "Starting container at $(date +%s.%N)"
START_TIME=$(date +%s.%N)

# Start the container in the background
CONTAINER_ID=$(docker run -d --gpus all \
    -p ${PORT}:${PORT} \
    ${IMAGE} \
    --model /models/meta-llama/Llama-3.1-8B-Instruct \
    --host 0.0.0.0 \
    --port ${PORT})

echo "Container ID: ${CONTAINER_ID}"

# Poll health endpoint until it responds
HEALTHY=false
while [ "$HEALTHY" = false ]; do
    if curl -sf http://localhost:${PORT}/health > /dev/null 2>&1; then
        HEALTHY=true
        HEALTH_TIME=$(date +%s.%N)
    else
        sleep 0.5
    fi
done

echo "Health check passed at $(date +%s.%N)"
COLD_START=$(echo "$HEALTH_TIME - $START_TIME" | bc)
echo "Cold start (to healthy): ${COLD_START}s"

# Now measure time to first inference
INFER_START=$(date +%s.%N)
curl -s http://localhost:${PORT}/v1/completions \
    -H "Content-Type: application/json" \
    -d '{"model": "meta-llama/Llama-3.1-8B-Instruct",
         "prompt": "Hello", "max_tokens": 1}' > /dev/null

INFER_TIME=$(date +%s.%N)
FIRST_INFER=$(echo "$INFER_TIME - $INFER_START" | bc)
TOTAL=$(echo "$INFER_TIME - $START_TIME" | bc)

echo "First inference latency: ${FIRST_INFER}s"
echo "Total cold start (to first token): ${TOTAL}s"

docker stop ${CONTAINER_ID} > /dev/null

Where the time goes

For a Llama 3.1 8B model in FP16 on an H100, with weights baked into the container image, the breakdown looks roughly like this:

# Typical cold start breakdown (Llama 3.1 8B, H100, baked weights)
#
# Phase                    Time
# Container init           ~2s
# Python + imports          ~5s
# CUDA context              ~3s
# Weight loading (disk)    ~12s   (16 GB from NVMe SSD)
# Weight transfer (GPU)     ~5s   (16 GB over PCIe Gen5: ~3.2 GB/s)
# KV cache allocation       ~1s
# CUDA graph capture       ~10s   (captures graphs for batch 1,2,4,8,16,32)
# --------------------------------
# Total                    ~38s

If weights are downloaded from a remote store instead of baked in, the weight loading phase jumps from 12 seconds to 30 to 90 seconds depending on network bandwidth.

Optimization strategies

Each phase has different optimization levers:

Weight loading. This is the biggest opportunity. Baking weights into the image (as I covered on day 53) eliminates download time. Using a local NVMe cache on the node eliminates it for subsequent starts. For the fastest possible loading, you can use tensorizer, a serialization format that supports streaming deserialization directly to GPU memory, cutting load time by 2 to 4x compared to safetensors.

# Using tensorizer for faster weight loading
# Serialize once:
python -m tensorizer.serialize \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --output /models/llama-8b.tensors

# Load in vLLM with tensorizer (supported since v0.5):
vllm serve /models/llama-8b.tensors \
    --load-format tensorizer

CUDA graph capture. You can reduce this by limiting the set of batch sizes for which graphs are captured. vLLM captures graphs for batch sizes that are powers of 2 up to max_num_seqs. Setting --max-num-seqs 32 instead of 256 reduces the number of graphs captured.

CUDA context. There is not much you can do about this, but NVIDIA's CUDA MPS (Multi-Process Service) can share a context across multiple processes, which helps if you are running multiple model servers on the same GPU via MIG.

Python imports. Using --worker-use-ray with pre-warmed Ray workers can amortize import time across container restarts, but this adds complexity.

Measuring in Kubernetes

In a Kubernetes deployment, cold start includes additional phases that the Docker measurement misses:

• Scheduling delay: Time for the kube-scheduler to find a node with a free GPU. This can be zero (GPU available) or minutes (waiting for a preemptible instance to spin up).
• Image pull: If the image is not cached on the node, pulling a 20 GB image takes 30 to 120 seconds depending on registry bandwidth.
• GPU driver initialization: The nvidia-container-toolkit needs to inject the GPU drivers. This adds 2 to 5 seconds.

# Measure Kubernetes cold start end-to-end
# Create the pod and watch events

kubectl apply -f vllm-pod.yaml
kubectl get events --watch --field-selector involvedObject.name=vllm-pod

# Look for these timestamps:
# Scheduled:     when the pod was assigned to a node
# Pulling:       image pull started
# Pulled:        image pull completed
# Started:       container process started
# Ready:         readiness probe passed (model loaded)

Cold start is not a single number. It is a pipeline, and optimizing it requires measuring each phase separately. The bottleneck is almost always weight loading, and the fix is almost always caching.

Next: round-robin and least-connections load balancers, because once you have multiple replicas running, you need to distribute traffic across them intelligently.