Building vLLM from source: a field guide

Building vLLM from source sounds like a pip install -e . away. In practice, on a fresh machine with a recent OS and a recent Python, you hit a chain of version-skew, driver, and toolchain issues that each fail with a cryptic message. This walks through a real end-to-end build on an AWS g5 instance (NVIDIA A10G) running Ubuntu 26.04 + Python 3.14, documenting every error encountered and the fix.

The working recipe

# 1. Confirm you actually have a GPU (see Pitfall 1)
lspci | grep -i nvidia        # hardware present?
nvidia-smi                    # driver working?

# 2. Driver (if nvidia-smi fails but lspci shows the GPU)
sudo apt-get install -y nvidia-driver-575-open nvidia-modprobe dkms

# 3. Virtual env
python3 -m venv ~/go/venv && source ~/go/venv/bin/activate
pip install --upgrade pip

# 4. CUDA torch + a CONSISTENT pip CUDA toolkit (one minor version)
pip install torch==2.11.0 torchvision==0.26.0 torchaudio==2.11.0
pip install "cuda-toolkit[nvcc]==13.3.0" "nvidia-cuda-runtime==13.3.29" \
            "nvidia-cuda-nvrtc==13.3.33" "nvidia-cublas==13.3.0.5"

# 5. Assemble CUDA_HOME from the pip layout
export CUDA_HOME=$VIRTUAL_ENV/lib/python3.*/site-packages/nvidia/cu13
ln -sfn $CUDA_HOME/lib $CUDA_HOME/lib64

# 6. Build (scope arch to YOUR GPU - A10G is sm_86)
export PATH=$CUDA_HOME/bin:$PATH CUDACXX=$CUDA_HOME/bin/nvcc
export VLLM_TARGET_DEVICE=cuda TORCH_CUDA_ARCH_LIST="8.6+PTX"
export MAX_JOBS=12 NVCC_THREADS=2
export CMAKE_ARGS="-DCUDAToolkit_ROOT=$CUDA_HOME -DCMAKE_CUDA_COMPILER=$CUDA_HOME/bin/nvcc"
pip install -v -e . --no-build-isolation

Pitfall 1: "There's no GPU here" when there definitely is

The very first check was nvidia-smi, which returned command not found. Conclusion drawn: no GPU, do a CPU build. Wrong. nvidia-smi missing only means the driver/userspace tools are not installed; it says nothing about the hardware. The actual hardware check is lspci:

$ lspci | grep -i nvidia
00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1)

The A10G was there the whole time; it just had no driver. On cloud instances that are not "Deep Learning AMIs," a bare GPU with no driver is the norm, not the exception. Lesson: lspci detects hardware. nvidia-smi detects a working driver. They answer different questions. Decide CPU-vs-GPU from lspci.

Pitfall 2: modprobe nvidia → "No such device"

The open-source nouveau driver grabs the GPU at boot. The NVIDIA module cannot bind while nouveau holds it. Fix: blacklist nouveau, unbind, and load:

echo -e "blacklist nouveau\noptions nouveau modeset=0" | \
    sudo tee /etc/modprobe.d/blacklist-nouveau.conf
echo -n "0000:00:1e.0" | sudo tee /sys/bus/pci/drivers/nouveau/unbind
sudo rmmod nouveau
sudo modprobe nvidia
sudo update-initramfs -u    # make the blacklist survive reboots

Pitfall 3: nvidia-smi works but CUDA returns error 999

This is the subtle one. After loading the module, nvidia-smi works and shows the A10G, but torch.cuda.is_available() throws "CUDA unknown error". A direct driver-API probe confirmed the runtime was broken even though nvidia-smi was fine:

import ctypes
ctypes.CDLL("libcuda.so.1").cuInit(0)   # → 999 (CUDA_ERROR_UNKNOWN)

Two distinct causes, both worth knowing:

Stale/incorrect UVM device nodes. nvidia-smi uses /dev/nvidia0 + /dev/nvidiactl (major 195). CUDA additionally needs /dev/nvidia-uvm. After a manual driver bring-up those nodes can be missing or have the wrong major. Recreate them against /proc/devices.
nvidia-modprobe is not installed. This setuid helper is what the CUDA runtime shells out to in order to create/initialize device nodes for non-root processes. Without it, raw cuInit may pass but torch's runtime init throws 999. This was the actual fix.

Lesson: nvidia-smi working does not equal CUDA working. They use different device nodes. If cuInit returns 999, look at /dev/nvidia-uvm and make sure nvidia-modprobe exists.

Pitfall 4: the nvidia-cuda-nvcc-cu13 package is a stub

The old naming is a trap. pip install nvidia-cuda-nvcc-cu13 only finds placeholder versions (0.0.0a0, 0.0.1). The real compiler ships via the cuda-toolkit[nvcc] extra (which pulls nvidia-cuda-nvcc, nvidia-nvvm, nvidia-cuda-crt). Use the meta package's extras, not the *-cu13 standalone names.

Pitfall 5: CUDA toolkit version skew

This was the single biggest time sink. The pip CUDA ecosystem is split across many packages and pip will happily install mismatched minor versions. Each mismatch fails differently:

ptxas can't assemble newer PTX: "ptxas fatal: Unsupported .version 9.3; current version is '9.0'".
nvcc and runtime disagree: cryptic link errors about missing symbols.
cublas and cudnn mismatch: runtime errors that only surface at inference time.

The fix is to pin every nvidia package to the same minor version. One minor version across the board.

Mental model

Building vLLM from source is like assembling a bicycle where every part is from a different year's model. The frame (Python), the wheels (torch), the brakes (CUDA toolkit) all mostly fit, but the tolerances are brutal and the error messages never point at the real problem.

Pitfall 6: a bundled submodule with its own Python check

Even after the main build succeeds, a bundled submodule can have its own narrower Python version check that fails at import time. The fix is usually a small patch to relax the check, or using the Python version the submodule expects.

Why this matters

Building vLLM from source is not a rite of passage; it is how you get the version you want, the kernels you need, and the ability to patch the engine. When you are serving at scale, the stock wheel is often not enough: you need a custom kernel, a backported fix, or an architecture-specific build. Knowing how to get from source to running engine, and knowing which errors are environmental versus real, is the difference between a morning of frustration and a clean build.

lspci detects hardware. nvidia-smi detects a working driver. They answer different questions. Decide CPU-vs-GPU from lspci.

Back to the blog