Small fine-tuned vs large quantized

This is one of those questions that comes up in every deployment conversation, and there is no universal answer. You have a task. You have a GPU budget. Do you take a small model (say, Llama 3.1 8B) and fine-tune it on your domain data? Or do you take a large model (Llama 3.1 70B), quantize it to INT4 to fit on fewer GPUs, and serve it zero-shot or with light prompting?

I have seen teams spend months on the wrong side of this tradeoff. Here is the framework I use to reason about it.

The compute profile is different

First, let us ground the comparison in hardware reality. Using the roofline framework:

• Llama 3.1 8B in FP16: 16 GB of weights. Fits on a single A100-80GB or H100 with plenty of room for KV cache. Decode is memory-bound, and you get high throughput because you are moving only 16 GB per decode step.
• Llama 3.1 70B in INT4 (GPTQ/AWQ): Roughly 35 GB of weights (70B params * 4 bits = 35 GB, plus some overhead for group scales). Also fits on a single 80 GB GPU, but with less room for KV cache. Decode moves 35 GB per step, so roughly half the tokens per second of the 8B model on the same hardware.

So the 8B model is about 2x faster at decode and uses half the memory. The question is whether it can match the quality of the quantized 70B.

When the small fine-tuned model wins

Fine-tuning shines when your task is narrow and well-defined. Some examples:

• Structured extraction: Pulling specific fields from documents in a known format. A fine-tuned 8B model with a few thousand examples will match or beat a prompted 70B model, because the task is learnable from examples and does not require broad world knowledge.
• Classification: Routing, sentiment, intent detection. These tasks are well-suited to fine-tuning, and small models learn them quickly.
• Domain-specific generation: If you need the model to write in a specific style, follow a rigid output schema, or use domain jargon correctly, fine-tuning encodes these patterns directly into the weights.
• High throughput, cost-sensitive: If you need to process millions of documents per day and your budget is fixed, the 8B model gives you 2x or more throughput per GPU dollar.

The general pattern: if you can collect a few thousand high-quality (input, output) pairs for your task, a fine-tuned small model is almost always the better deployment choice. The model learns a compressed version of the task, and you pay less per inference.

When the large quantized model wins

Quantized large models win when the task requires broad knowledge or complex reasoning that a small model simply lacks the capacity for:

• Open-ended reasoning: Multi-step logic, math, code generation. The 70B model has seen more data and has more parameters to store reasoning patterns. INT4 quantization costs roughly 1 to 2 points on benchmarks like MMLU, which is usually worth the deployment savings.
• Few-shot generalization: When you do not have fine-tuning data, or your task changes frequently, the large model's in-context learning is more robust.
• Long-tail knowledge: If your task requires knowing obscure facts, the larger model's training set coverage matters.
• Low data availability: If you have fewer than 100 examples, fine-tuning a small model is risky. You are likely to overfit or learn spurious patterns. The large model with careful prompting is safer.

The quality/cost frontier

Here is how I think about the tradeoff quantitatively. For any given task, you can plot quality (your eval metric) against cost (dollars per 1M tokens). The frontier looks something like this:

Quality
  ^
  |          * 70B FP16 (2 GPUs)
  |        * 70B INT4 (1 GPU)
  |      * 8B fine-tuned
  |    * 8B base
  |  * 3B fine-tuned
  +-------------------------> Cost ($/1M tokens)
  low cost                    high cost

The fine-tuned small model often sits at a surprisingly good position on this curve: it can match the quality of the quantized large model at a fraction of the cost, if you have the data to fine-tune it. When it does not match, the gap is usually in edge cases and tail distributions that matter for some applications and not others.

The hybrid approach

In practice, many production systems use both. A common pattern is routing:

• A small fine-tuned model handles the 80% of requests that are routine and well-covered by training data.
• A large model (quantized or full precision) handles the remaining 20% that require more reasoning or fall outside the small model's comfort zone.
• A lightweight classifier or confidence threshold decides which model to use.

This gives you the throughput and cost advantages of the small model for most traffic, with the quality backstop of the large model for hard cases. The cache-aware routing ideas apply here too: route to the model that already has relevant context cached.

Quantization quality in practice

Not all quantization is equal, and this matters for the comparison. From my experience with INT8 quantization and GPTQ:

• INT8 (W8A8): Essentially lossless for most models above 7B parameters. If your only constraint is memory, INT8 is almost free quality.
• INT4 (GPTQ, AWQ): Typically 1 to 3 points of degradation on benchmarks like MMLU for 70B models. For specific tasks, the degradation can be higher or lower. Always measure on your task.
• INT4 with group quantization (group size 128): Better than per-channel INT4, adding about 10% memory overhead for the group scales but recovering most of the quality loss.
• NVFP4 (Blackwell): NVIDIA's 4-bit floating point format with hardware support. Early results suggest it is better than INT4 for the same bit width because the floating-point representation handles outliers more gracefully.

The point is that a "70B INT4" is not one thing. The specific quantization method, calibration data, and group size all affect quality. If you are comparing against a fine-tuned 8B, you need to quantize carefully and measure on your actual task.

Fine-tuning costs and risks

Fine-tuning is not free. Beyond the GPU hours for training, the real costs are:

• Data collection and cleaning: You need high-quality (input, output) pairs. This is often the bottleneck, not the training itself.
• Evaluation: You need a robust eval harness to know whether your fine-tune actually improved things.
• Maintenance: Fine-tuned models drift as the world changes. You need a pipeline to retrain periodically.
• Catastrophic forgetting: Fine-tuning on a narrow task can degrade the model's general capabilities. If your users sometimes ask questions outside the fine-tuning distribution, the model may perform worse than the base model on those queries.

A distilled model can help here: instead of fine-tuning on human-labeled data, you distill from the large model's outputs on your task distribution, getting some of the quality of the large model at the inference cost of the small one.

My decision framework

Start with the large model prompted for your task. Measure quality and cost. If quality is sufficient and cost is acceptable, ship it. If cost is too high, try quantizing. If quality degrades too much, fine-tune a small model. If you lack data, distill from the large model.

The mistake I see most often is teams jumping straight to fine-tuning without establishing a baseline with the large model. You need to know what "good" looks like before you can optimize for cost. And you need an eval to know whether your optimization worked.

Next: distillation for inference, the technique that lets you transfer knowledge from a large teacher to a small student without human-labeled data.