Multi-GPU instances and MIG

One of the first things that surprised me about production inference was how much time I spent thinking about instance types. The model does not care whether it is running on AWS, GCP, or bare metal. But the instance shape determines how many GPUs you have, how they are connected, how much CPU memory is available, and ultimately what your cost-per-token looks like.

Multi-GPU instance topologies

Cloud providers package GPUs into instances with specific interconnect topologies. The topology matters because it determines which parallelism strategies are efficient:

• Single GPU (e.g., AWS p5.xlarge equivalent, GCP a3-highgpu-1g): one H100. Good for models up to ~35B in FP8. No parallelism overhead. Simplest to operate.
• 2-GPU: two GPUs typically connected by NVLink. TP=2 is efficient here. Fits models up to ~70B in FP8.
• 4-GPU: four GPUs with NVLink mesh. TP=4 works well. Good for 70B models with generous KV cache budgets.
• 8-GPU (e.g., AWS p5.48xlarge, GCP a3-megagpu-8g): the full DGX-like configuration. 8 H100s connected through NVSwitch with 900 GB/s all-to-all bandwidth. This is where TP=8 shines. Required for models like Llama 3.1 405B (which needs at minimum TP=8 in FP8).

The critical detail is the interconnect. Within an 8-GPU instance, all GPUs communicate through NVSwitch at 900 GB/s. If you try to do TP across two separate instances connected by network (even 400 Gbps InfiniBand), you drop to roughly 50 GB/s, an 18x slowdown in communication. This is why pipeline parallelism is used across nodes while tensor parallelism stays within a node.

MIG: Multi-Instance GPU

NVIDIA Multi-Instance GPU (MIG), introduced with the A100 and supported on H100, allows you to partition a single physical GPU into up to seven isolated instances. Each MIG instance gets a dedicated slice of:

• SMs: a fixed number of compute units
• HBM: a dedicated portion of memory, with hardware-enforced isolation
• Memory bandwidth: a proportional share of the total bandwidth
• L2 cache: a dedicated slice

On the H100, the supported MIG profiles include:

# H100 MIG profiles (SXM variant)
# Profile         SMs    Memory    Bandwidth (approx)
# 1g.10gb         16     10 GB     ~480 GB/s
# 2g.20gb         32     20 GB     ~960 GB/s
# 3g.40gb         48     40 GB     ~1.6 TB/s
# 4g.40gb         64     40 GB     ~1.6 TB/s
# 7g.80gb         132    80 GB     ~3.35 TB/s (full GPU)

# Create a MIG instance
sudo nvidia-smi mig -cgi 3g.40gb -C
# List MIG instances
nvidia-smi mig -lgi

The isolation is hardware-enforced, not software. One MIG instance cannot see another's memory, cannot interfere with its compute, and cannot cause it to OOM. This is stronger isolation than running two processes on the same GPU (which share the same memory pool and can interfere).

When MIG makes sense for inference

MIG is valuable in a few specific scenarios:

• Small models on large GPUs: if you are serving a 7B model in FP8 (~3.5 GB weights), using a full H100 is wasteful. A 1g.10gb or 2g.20gb MIG slice is more than sufficient, and you can run 3 to 7 independent model instances on the same physical GPU.
• Multi-tenant environments: when different teams or customers need guaranteed GPU resources without interference. Each tenant gets a MIG slice with guaranteed compute and memory.
• Mixed workloads: run a latency-sensitive chat model on one MIG slice and a throughput-oriented batch processing model on another, on the same GPU, without contention.

Right-sizing your deployment

Here is how I think about instance selection for inference:

1. Start with the model size. How many bytes of weights, plus KV cache budget, plus activation memory? This gives you the minimum HBM.
2. Check if it fits on one GPU. If yes, use a single-GPU instance or a MIG slice. Simpler is better.
3. If it needs multiple GPUs, determine TP degree. TP should stay within NVLink-connected GPUs. For most models, TP=2, 4, or 8 within a single node.
4. If it needs more than 8 GPUs, add PP across nodes. This is where the infrastructure cost jumps significantly.
5. Consider MIG for small models. If you are running a 7B model and paying for a full H100, you are leaving 70+ GB of HBM and most of the compute idle.

A practical example: serving Llama 3 8B in FP8 (~4 GB weights). On a full H100, this uses 5% of the memory. With MIG, you could run it on a 1g.10gb slice and fit 7 independent instances on one GPU. At $30/hour for a full H100, that is roughly $4.30/hour per model instance instead of $30. The cost difference is substantial.

Kubernetes and GPU sharing

In Kubernetes, MIG instances appear as separate GPU resources. The NVIDIA device plugin exposes them as distinct resource types:

# Pod requesting a MIG slice
resources:
  limits:
    nvidia.com/mig-1g.10gb: 1

# vs. a full GPU
resources:
  limits:
    nvidia.com/gpu: 1

This integrates cleanly with Kubernetes scheduling: the scheduler treats MIG slices as independent resources, and pods requesting a MIG slice are guaranteed hardware isolation from other pods on the same GPU.

The goal of infrastructure sizing is not to use the biggest GPU available. It is to use the smallest GPU (or slice) that meets your latency and throughput requirements. Everything else is wasted cost.

Next: containers and NIMs, where we look at how inference workloads get packaged and deployed, and what NVIDIA's NIM containers bring to the table.