Skand Tandon
Modalities

Multi-modal batching

When your serving system handles text, images, audio, and video in the same queue, naive batching breaks down. Each modality has different token counts, preprocessing costs, and memory profiles. Here is how to build a batch scheduler that handles them all.

2025-12-24 modalities 7 min read

The moment you deploy a vision-language model (VLM) like LLaVA, Qwen-VL, or GPT-4o, your neat text-only batching logic falls apart. A text-only request might contribute 200 tokens to a batch. An image request might contribute 200 text tokens plus 576 image tokens (a 384x384 image at patch size 16). A video request could add thousands more. Suddenly, "batch size 32" means wildly different things depending on what is in the batch.

I spent a couple of days working through how modern serving systems handle this, and the answer is surprisingly principled once you see it.

The core problem: heterogeneous token counts

In a text-only system, you can reason about batching in terms of request count or total token count. Continuous batching systems like vLLM and SGLang schedule based on the total number of tokens that fit in the KV cache and GPU memory. This works because every token is the same size: one embedding vector of dimension d_model.

Multi-modal requests break this assumption in two ways:

Strategy 1: token budget batching

The simplest approach, and the one that works well in practice, is to batch by total token budget rather than request count. You set a maximum total token count for the batch (say, 8,192 prefill tokens) and greedily add requests from the queue until the next request would exceed the budget.

def form_batch(queue, max_tokens=8192):
    batch = []
    total = 0
    for req in queue:
        req_tokens = req.text_tokens + req.image_tokens + req.audio_tokens
        if total + req_tokens > max_tokens and len(batch) > 0:
            break
        batch.append(req)
        total += req_tokens
    return batch

This naturally handles the heterogeneity: a batch might contain 30 text-only requests (200 tokens each) or 3 image requests (2,500 tokens each). The GPU memory usage is roughly the same either way because what matters is the total number of tokens flowing through the transformer.

Why this works

The transformer does not care whether a token came from text, an image patch, or an audio frame. Once the modality encoder has projected everything into the shared embedding space, every token is just a vector of dimension d_model. The attention computation, the FFN, and the KV cache all scale with total token count, not with request count or modality.

Strategy 2: async preprocessing pipelines

The preprocessing bottleneck is real. If you run the ViT encoder and Whisper encoder inline with batch formation, you introduce variable latency that stalls the GPU. The solution is to decouple preprocessing from inference:

This pipeline structure means the LLM GPU never waits on image encoding. Preprocessing and inference overlap in time. SGLang and vLLM both implement variants of this pattern for their VLM support.

Handling dynamic resolution

Modern VLMs like Qwen2-VL and InternVL2 support dynamic resolution: instead of resizing every image to a fixed size, they tile the image into multiple crops and encode each crop separately. A small icon might become 1 tile (256 tokens), while a detailed photograph might become 12 tiles (3,072 tokens).

This makes batch formation harder because you do not know the exact token count until you have decided on the tiling. The practical approach is:

def estimate_image_tokens(image, max_tiles=12):
    """Estimate token count based on image resolution."""
    h, w = image.size
    # Find optimal tiling that respects aspect ratio
    tiles_h = min(ceil(h / tile_size), max_tiles_h)
    tiles_w = min(ceil(w / tile_size), max_tiles_w)
    n_tiles = min(tiles_h * tiles_w, max_tiles)
    tokens_per_tile = (tile_size // patch_size) ** 2  # e.g., 256
    return n_tiles * tokens_per_tile + n_tiles  # +1 per tile for separator

Run this estimation at request admission time, before full preprocessing, so the scheduler has an accurate token budget to work with.

The KV cache complication

Image and audio tokens participate in prefill but typically do not generate output tokens during decode. The text tokens that follow them do. This creates an asymmetry in KV cache management:

One optimization is visual token compression: after the cross-attention layers that fuse visual information into the text representation, you can drop or merge visual KV entries that the model no longer attends to heavily. Techniques like FastV identify and prune low-attention visual tokens after the first few layers.

Practical batching configurations

Here is what I have seen work well for different deployment scenarios:

The unifying principle: batch on total tokens, not request count. Preprocess asynchronously. Let the transformer see a flat sequence regardless of where the tokens came from.

Tomorrow we build an embedding similarity search pipeline, where batching matters for throughput but the model output is a single vector per input rather than a sequence of generated tokens.