Embedding model inference at scale

After spending weeks on autoregressive LLMs, I found embedding model inference almost refreshing. No KV cache. No speculative decoding. No continuous batching headaches. You take a piece of text, run it through a transformer encoder, and get back a fixed-size vector. Simple, right?

It is, until you need to serve a few thousand requests per second with p99 latency under 20 milliseconds. Then things get interesting fast.

What makes embeddings different from LLMs

An embedding model like E5, BGE, or GTE is typically a BERT-style encoder. The forward pass is a single shot: you feed in a sequence of tokens, every layer runs once, and you extract a vector from the final hidden states. There is no autoregressive loop, no token-by-token generation, no KV cache that grows with sequence length.

This changes the performance profile completely:

• Compute-bound, not memory-bound. Because there is no decode phase, there is no memory-bandwidth bottleneck from loading weights per token. The entire workload is a series of dense matrix multiplications, which means we can actually saturate the GPU's compute units.
• Throughput is the game. Embedding requests are cheap individually but come in massive volumes. A retrieval system might embed every user query plus every new document chunk. At scale, that is millions of calls per day.
• Latency is tight. Embeddings sit in the critical path of search: the user types a query, you embed it, and then you search. Every millisecond in the embedding step delays the search result.

The padding problem

The first surprise when serving embeddings at scale is how much time you waste on padding. BERT-style models need fixed-length tensors within a batch. If your batch has sequences of length 12, 45, and 510, the naive approach pads everything to 512 tokens. That means the 12-token sequence burns 500 tokens worth of compute doing nothing useful.

There are two main approaches to fix this:

• Sorted batching. Sort incoming requests by sequence length and batch similar lengths together. A batch of [12, 14, 15, 18] tokens wastes far less padding than [12, 45, 510, 8]. This is surprisingly effective and easy to implement with a small reorder buffer.
• Sequence packing (unpadding). Concatenate all sequences into one long tensor and use a position index to track boundaries. HuggingFace Transformers supports this via unpad_inputs in some models. FlashAttention handles it natively with the cu_seqlens interface.

# Sequence packing with FlashAttention
# Instead of padding to max_len:
# input_ids: [B, max_len] with lots of padding

# Pack sequences end-to-end:
# packed_ids: [total_tokens]  (no padding)
# cu_seqlens: [0, len1, len1+len2, ...]  (cumulative lengths)

from flash_attn import flash_attn_varlen_func

output = flash_attn_varlen_func(
    q, k, v,
    cu_seqlens_q=cu_seqlens,
    cu_seqlens_k=cu_seqlens,
    max_seqlen_q=max_len,
    max_seqlen_k=max_len,
)

With sequence packing, I have seen throughput improvements of 2x to 4x on workloads with highly variable sequence lengths. The improvement scales with how skewed your length distribution is.

Pooling: mean, CLS, or last token

After the forward pass, you need to reduce the per-token hidden states into a single vector. The choice of pooling strategy matters more than most people realize, because it interacts with how the model was trained:

• CLS pooling takes the hidden state of the [CLS] token. Classic BERT approach, but many modern embedding models are not trained with CLS pooling and will give poor results if you use it.
• Mean pooling averages all token hidden states, weighted by the attention mask to exclude padding. This is the most common strategy for models like E5 and BGE.
• Last-token pooling takes the hidden state of the final real token. Used by some decoder-based embedding models like Mistral's E5-mistral.

Getting this wrong is a silent killer. The embeddings will still be 1024-dimensional vectors that look reasonable, but retrieval quality will be quietly degraded. Always check the model card for the correct pooling strategy.

Normalization and similarity

Most modern embedding models are trained with cosine similarity, which means the vectors should be L2-normalized before storage and search. This seems trivial, but it has a practical implication for serving: you should normalize on the server side, not the client side. This way every downstream consumer gets correctly normalized vectors, and you avoid a class of bugs where one client forgets to normalize and gets garbage similarity scores.

# Server-side normalization
import torch.nn.functional as F

embeddings = model(input_ids, attention_mask)
embeddings = mean_pool(embeddings, attention_mask)
embeddings = F.normalize(embeddings, p=2, dim=-1)  # L2 normalize

# Now cosine similarity == dot product
# which is what vector databases optimize for

Quantization and distillation

Because embedding models are compute-bound (not memory-bound like LLM decode), quantization helps differently here. INT8 quantization of an embedding model does not primarily reduce memory bandwidth pressure. Instead, it doubles the throughput of matrix multiplications on Tensor Cores that support INT8 (A100, H100). On an H100, INT8 gives roughly 2x the TOPS compared to FP16.

The quality impact depends heavily on the model. I have seen models where INT8 dynamic quantization causes less than 0.1% degradation in retrieval recall, and others where it drops by 2 to 3%. Always measure on your actual retrieval benchmark before deploying quantized embeddings in production.

For even higher throughput, consider using a smaller distilled model. A well-distilled 33M parameter model can match a 330M parameter model on many benchmarks while running 10x faster. Models like bge-small-en-v1.5 (33M params) versus bge-large-en-v1.5 (335M params) illustrate this tradeoff well.

Batching strategy for production

The optimal serving setup for embedding models differs from LLMs. Here is what I have found works well:

• Use dynamic batching with a short timeout (5 to 10 ms) and a generous max batch size (64 to 256). Embedding forward passes are fast enough that you can afford to wait a few milliseconds to fill a bigger batch.
• Set max sequence length aggressively. If your documents are chunked to 512 tokens, do not accept 8192-token inputs. Truncate or reject early.
• Separate query and document endpoints. Queries are short (10 to 30 tokens) and latency-sensitive. Documents are longer and throughput-sensitive. Different batching parameters for each makes a real difference.
• Use ONNX Runtime or TensorRT for the forward pass. Graph-level optimizations like layer fusion and constant folding give 30 to 50% speedups over eager PyTorch for encoder models.

Embedding inference is the rare case where "just throw a bigger GPU at it" is often the wrong answer. A well-optimized pipeline on a T4 can outperform a naive setup on an A100, because the bottleneck is almost always in the batching and padding, not raw compute.

Connection to vector search

The embedding model is only half the story. What you do with the vectors afterward matters just as much. The choice of embedding dimension (384, 768, 1024, or higher) directly affects vector index size and search speed. Some models like Nomic Embed support Matryoshka representation learning, where you can truncate the vector to a shorter dimension (say 256 instead of 768) with minimal quality loss. This halves your index memory and speeds up search.

If you are building an embedding pipeline, think about the full loop: model size, quantization, pooling, normalization, dimension, index type. Each choice compounds. Get them all right and you can serve millions of queries per day on a single GPU. Get one wrong and you will be throwing hardware at a software problem.

Next up: Whisper ASR, where the encoder-decoder architecture brings back some of the complexity we just avoided.