Skand Tandon
Modalities

VLM inference: image preprocessing and batching

Vision-language models look like text models with extra steps, but those extra steps change the inference picture dramatically. Image decoding, resizing, patch embedding, and variable-length visual token sequences all create new bottlenecks. Here is what I learned serving them.

2025-12-18 modalities 8 min read

After spending weeks on text-only LLM inference, serving my first vision-language model felt like starting over. The text side is familiar: tokenize, prefill, decode, stream. But images add an entire preprocessing pipeline that runs before the language model even starts, and it introduces bottlenecks in places I did not expect.

Today I want to walk through the full VLM inference pipeline, focusing on the image side: how images get turned into tokens, what makes batching harder, and where the performance traps are.

The VLM architecture in brief

Most modern VLMs (LLaVA, Qwen-VL, InternVL, Llama 3.2 Vision) follow the same basic pattern. An image encoder (usually a Vision Transformer like CLIP ViT or SigLIP) processes the image into a sequence of visual embeddings. A projection layer maps these embeddings into the language model's embedding space. The visual tokens are then interleaved with text tokens and fed to the language model as if they were part of the text sequence.

The language model itself does not know it is looking at an image. It sees a sequence of embeddings, some from text tokens and some from visual tokens, and runs the same transformer forward pass. The magic is all in the preprocessing.

# Simplified VLM forward pass
def vlm_forward(image, text_tokens, vision_encoder, projector, llm):
    # Step 1: Image preprocessing (CPU)
    pixel_values = preprocess_image(image)  # resize, normalize

    # Step 2: Vision encoding (GPU)
    visual_features = vision_encoder(pixel_values)  # ViT forward

    # Step 3: Projection (GPU)
    visual_tokens = projector(visual_features)  # map to LLM space

    # Step 4: Merge with text tokens
    input_embeds = merge_tokens(visual_tokens, text_tokens)

    # Step 5: LLM forward (GPU)
    output = llm(inputs_embeds=input_embeds)
    return output

Image preprocessing: the CPU bottleneck

Before the image reaches the GPU, it has to go through several CPU-bound steps:

For a single image, the total preprocessing takes 10 to 30 ms. That does not sound like much, but it is entirely CPU-bound, and in a serving context, it can become a bottleneck. If your server receives 100 image requests per second, you need 1 to 3 full CPU cores just for image preprocessing.

Performance tip

Replace Pillow with Pillow-SIMD or use torchvision's decode_jpeg with the CUDA backend for GPU-accelerated JPEG decoding. On an H100, GPU JPEG decoding can process 500+ images per second, compared to 100 to 200 on CPU with Pillow. The NVIDIA DALI library is another option for building a fully GPU-accelerated preprocessing pipeline.

Vision encoding: the prefill within prefill

The vision encoder is typically a ViT-L/14 (307M parameters) or ViT-H/14 (632M parameters). For a 336x336 image with 14x14 patches, this produces 576 patch tokens (24x24 patches). Each patch goes through the full ViT transformer stack.

The vision encoder forward pass takes 5 to 15 ms on an H100 for a single image with ViT-L. With dynamic resolution and 6 tiles, that becomes 30 to 90 ms, which is comparable to or longer than the language model prefill for a medium-length text prompt.

This creates an interesting scheduling challenge. The vision encoder runs synchronously before the language model can start. For text-only requests in the same batch, the language model sits idle while the vision encoder processes images for other requests. Smart scheduling interleaves text-only requests during vision encoding to keep the LLM busy.

Variable-length visual token sequences

This is where batching gets tricky. Text-only models produce a fixed number of tokens per input token (one to one). But the number of visual tokens per image varies:

Variable-length sequences mean you cannot simply stack visual tokens into a uniform tensor for the language model. You need to handle padding or use packed sequences. vLLM handles this by treating visual tokens like variable-length prefill sequences and using its existing paged attention infrastructure. SGLang takes a similar approach with RadixAttention.

# Token count varies by image size (Qwen2-VL example)
def estimate_visual_tokens(width, height, patch_size=14, min_pixels=256*28*28):
    # Qwen2-VL scales tokens with image resolution
    pixels = max(width * height, min_pixels)
    tokens_h = height // patch_size
    tokens_w = width // patch_size
    # Temporal merging reduces by 2x2
    return (tokens_h // 2) * (tokens_w // 2)

# 336x336 -> 144 tokens
# 672x672 -> 576 tokens
# 1344x672 -> 1152 tokens

Batching strategy for VLMs

The ideal batching strategy for VLMs differs from text-only models. I have found these principles useful:

Multi-image requests

Some VLM use cases involve multiple images per request: compare these two images, describe this document page by page, analyze this sequence of frames. Each additional image multiplies the visual token count and the preprocessing time.

A request with 10 images at 576 tokens each adds 5760 visual tokens, equivalent to a 5760-token text prefix in terms of KV cache consumption and prefill compute. This can easily dominate the batch's memory budget and crowd out other requests.

The practical limit I have found is around 4 to 8 images per request before latency becomes unacceptable for interactive use. For batch processing (document understanding, video frame analysis), you can handle more images but should stream the results and process pages sequentially rather than loading all images at once.

VLM inference is not just LLM inference with images bolted on. The variable token counts, CPU preprocessing bottleneck, and dual-model architecture create fundamentally different performance characteristics. Profile the full pipeline, not just the language model, or you will optimize the wrong thing.

For background on how the language model side works, see ops:byte ratio and the roofline. For multi-modal batching strategies across modalities, see multi-modal batching. And for the KV cache management that visual tokens consume, check KV cache manager: block allocator, eviction.