Skand Tandon
Production Systems

Configure MIG on a GPU

Multi-Instance GPU lets you carve one A100 or H100 into up to seven isolated GPU instances. I walked through the setup, the partition profiles, and when MIG makes sense for inference workloads.

2025-11-30 production systems 7 min read

Yesterday I benchmarked tensor parallelism, which spreads one large model across multiple GPUs. Today I went in the opposite direction: using NVIDIA's Multi-Instance GPU (MIG) to carve one physical GPU into multiple isolated instances, each with its own compute resources, memory, and memory bandwidth.

MIG is available on A100, A30, H100, and H200 GPUs. The idea is straightforward: if your model is small enough to fit in a fraction of the GPU's memory, you are wasting expensive silicon by giving it the whole device. MIG lets you pack multiple models (or multiple replicas of the same model) onto one physical GPU with hardware-level isolation.

How MIG works

A GPU like the A100 has 108 streaming multiprocessors (SMs) and 80 GB of HBM2e. MIG partitions these resources into GPU Instances (GIs), each of which gets a dedicated slice of SMs, memory, and L2 cache. Within each GI, you create one or more Compute Instances (CIs) that further subdivide the SMs.

The key property is hardware isolation. Unlike time-slicing (MPS) or CUDA context switching, MIG instances have physically separate memory and compute. A noisy neighbor in one instance cannot affect the latency or throughput of another. This is critical for inference SLOs where tail latency matters.

The A100 80GB supports these GPU Instance profiles:

The H100 has similar profiles but with its 132 SMs and different memory configurations. The naming convention is <compute_slices>g.<memory>gb.

Setting it up

MIG configuration requires root and a GPU reset. Here is the step-by-step process I followed on an A100:

# Step 1: Enable MIG mode on GPU 0
sudo nvidia-smi -i 0 -mig 1

# Step 2: Reboot or reset the GPU
# (In practice, a driver reload works)
sudo nvidia-smi -i 0 -r

# Step 3: List available GPU Instance profiles
nvidia-smi mig -i 0 -lgip

# Step 4: Create GPU Instances
# Example: create 3 instances of 2g.20gb
sudo nvidia-smi mig -i 0 -cgi 14,14,14

# Step 5: Create Compute Instances within each GI
sudo nvidia-smi mig -i 0 -cci

# Step 6: Verify the configuration
nvidia-smi mig -i 0 -lgi

The profile ID (14 in the example) comes from the -lgip output. Each profile has a numeric ID. After creation, nvidia-smi shows the MIG instances as separate devices:

# Each MIG instance appears as a separate CUDA device
nvidia-smi
# GPU 0: A100 80GB
#   MIG 2g.20gb  Device 0: UUID MIG-xxxx-0
#   MIG 2g.20gb  Device 1: UUID MIG-xxxx-1
#   MIG 2g.20gb  Device 2: UUID MIG-xxxx-2

Applications target a specific MIG instance via the CUDA_VISIBLE_DEVICES environment variable using the MIG UUID:

# Run vLLM on a specific MIG instance
CUDA_VISIBLE_DEVICES=MIG-xxxx-0 python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-2-7b-hf \
  --dtype float16 \
  --max-model-len 2048 \
  --gpu-memory-utilization 0.9

When MIG makes sense for inference

MIG is most useful when you are serving small to medium models that do not need a full GPU. The canonical use case is running multiple 7B parameter models (or smaller) on a single A100 or H100.

MIG vs. time-slicing

Kubernetes and NVIDIA's device plugin support both MIG and time-slicing for GPU sharing. MIG gives hardware isolation and predictable performance but limits you to fixed partition sizes. Time-slicing (via MPS or default CUDA scheduling) is more flexible but has no isolation: one workload's memory pressure or compute burst affects all others. For production inference with SLOs, MIG is almost always the better choice.

MIG in Kubernetes

In a Kubernetes cluster, NVIDIA's GPU operator and device plugin can expose MIG instances as schedulable resources. You configure the device plugin with a strategy:

# nvidia-device-plugin ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: nvidia-device-plugin
data:
  config.yaml: |
    version: v1
    sharing:
      mig:
        strategy: single
        resources:
          - name: nvidia.com/mig-2g.20gb
            replicas: 3

Pods then request specific MIG profiles:

resources:
  limits:
    nvidia.com/mig-2g.20gb: 1

This is cleaner than manually setting CUDA_VISIBLE_DEVICES and works well with autoscalers. The GPU operator handles creating and destroying MIG instances as the cluster scales.

Performance characteristics

I ran a quick benchmark serving Llama 2 7B on a single A100 in three configurations: full GPU, 4g.40gb MIG instance, and 2g.20gb MIG instance. Using the same request workload (100 ShareGPT prompts, concurrency of 8):

The performance scales roughly linearly with SM count. Memory bandwidth scales similarly because MIG partitions the HBM stacks. A 2g.20gb instance gets about 1/4 of the total memory bandwidth, not just 1/4 of the memory capacity.

The economics are what matter: three 2g.20gb instances serving three replicas give you about 84% of the throughput of three separate full GPUs, but on one physical GPU. If each replica serves a different model or customer, you have tripled your model density without buying more hardware.

Gotchas

MIG is a powerful tool for right-sizing GPU allocation when your models are small enough to share. Combined with the blue-green deployment patterns I will explore next, it enables high-density, low-waste inference clusters.