Skand Tandon
Advanced

Build an eval harness for a deployed model

You cannot improve what you do not measure. A practical eval harness hits your live endpoint, scores responses, tracks regressions, and tells you whether that quantization or fine-tune actually helped.

2026-01-06 advanced 9 min read

Every time I talk about choosing between model sizes or distillation, the conversation ends the same way: "how do I know if it actually works?" The answer is an eval harness. Not a Jupyter notebook you run once. A repeatable, automated pipeline that you can point at any endpoint and get back a score you trust.

This is the post where I build one from scratch. Not a framework review, not a comparison of lm-eval-harness versus HELM. Just the practical components you need to evaluate a deployed model behind an OpenAI-compatible API.

The architecture

An eval harness has four components:

  1. Test suite: A set of (input, expected_output, scoring_function) triples.
  2. Runner: Sends inputs to the model endpoint and collects responses.
  3. Scorer: Compares responses to expected outputs and produces metrics.
  4. Reporter: Aggregates scores and presents results in a way that supports decisions.

Let me walk through each one.

Test suite design

The test suite is the hardest part to get right and the most important. I structure mine as JSONL files, one per task category:

# eval_data/extraction.jsonl
{"id": "ext-001", "input": "Extract the company name and revenue from: 'Acme Corp reported Q3 revenue of $4.2B'", "expected": {"company": "Acme Corp", "revenue": "$4.2B"}, "scorer": "json_match"}
{"id": "ext-002", "input": "Extract the company name and revenue from: 'FY2024 results: GlobalTech, revenue 890M EUR'", "expected": {"company": "GlobalTech", "revenue": "890M EUR"}, "scorer": "json_match"}

# eval_data/reasoning.jsonl
{"id": "rea-001", "input": "If a GPU has 80GB HBM and the model weights are 35GB, and each KV cache entry per layer is 512 bytes with 80 layers, how many tokens of KV cache can fit?", "expected_contains": ["562500", "562,500"], "scorer": "contains_any"}

A few principles I follow:

The runner

The runner is straightforward: async HTTP requests to the model endpoint. I use asyncio with aiohttp for concurrency control:

import asyncio
import aiohttp
import json
import time

class EvalRunner:
    def __init__(self, endpoint: str, api_key: str, model: str,
                 concurrency: int = 10):
        self.endpoint = endpoint
        self.api_key = api_key
        self.model = model
        self.semaphore = asyncio.Semaphore(concurrency)

    async def run_single(self, session, test_case):
        async with self.semaphore:
            start = time.monotonic()
            async with session.post(
                f"{self.endpoint}/v1/chat/completions",
                headers={"Authorization": f"Bearer {self.api_key}"},
                json={
                    "model": self.model,
                    "messages": [{"role": "user",
                                  "content": test_case["input"]}],
                    "temperature": 0,
                    "max_tokens": 1024,
                }
            ) as resp:
                result = await resp.json()
                elapsed = time.monotonic() - start

            return {
                "id": test_case["id"],
                "input": test_case["input"],
                "output": result["choices"][0]["message"]["content"],
                "latency_s": elapsed,
                "tokens": result.get("usage", {}),
            }

    async def run_suite(self, test_cases):
        async with aiohttp.ClientSession() as session:
            tasks = [self.run_single(session, tc) for tc in test_cases]
            return await asyncio.gather(*tasks)

Key choices here:

Scoring functions

Different tasks need different scorers. Here is a minimal set that covers most use cases:

import json
import re

def exact_match(output: str, expected: str) -> float:
    return 1.0 if output.strip() == expected.strip() else 0.0

def contains_any(output: str, expected_contains: list) -> float:
    return 1.0 if any(s in output for s in expected_contains) else 0.0

def json_match(output: str, expected: dict) -> float:
    try:
        # Extract JSON from markdown code blocks if present
        match = re.search(r'```json?\s*(.*?)```', output, re.DOTALL)
        parsed = json.loads(match.group(1) if match else output)
    except (json.JSONDecodeError, AttributeError):
        return 0.0

    # Score: fraction of expected keys with correct values
    correct = sum(
        1 for k, v in expected.items()
        if parsed.get(k, "").strip().lower() == str(v).strip().lower()
    )
    return correct / len(expected)

def llm_judge(output: str, input_text: str, criteria: str,
              judge_endpoint: str) -> float:
    """Use a separate LLM to score the response on a 1-5 scale."""
    prompt = f"""Rate the following response on a scale of 1-5 based on: {criteria}

Input: {input_text}
Response: {output}

Reply with only a number from 1 to 5."""

    # Call judge endpoint (implementation similar to runner)
    score = call_judge(judge_endpoint, prompt)
    return float(score) / 5.0  # normalize to 0-1

The LLM-as-judge scorer deserves special attention. It is powerful but noisy. I recommend:

The reporter

Scores are only useful in context. The reporter should show:

# Example report output
======================================
Eval Report: llama-3.1-8b-ft-v3
Endpoint:    https://api.example.com/v1
Date:        2026-01-06
Test Suite:  v2.4 (312 examples)
======================================

Task              N    Score   Prev    Delta
-----------      ---   -----   -----   ------
extraction        80   0.94    0.91    +0.03
reasoning         50   0.78    0.76    +0.02
summarization     62   0.82    0.84    -0.02  ⚠️
code_gen          70   0.71    0.72    -0.01
style             50   0.88    0.85    +0.03
-----------      ---   -----   -----   ------
overall          312   0.83    0.82    +0.01

Latency: P50=0.8s  P95=2.1s  P99=4.3s
Throughput: 45 tok/s avg

⚠️ Regression detected in: summarization (-0.02)

The delta column is the whole point. Without a comparison against the previous model version, a score of 0.82 means nothing. With the delta, you can make decisions: the fine-tune improved extraction and reasoning but slightly hurt summarization. Is that tradeoff acceptable?

Ship rule

I never ship a model change without running the eval harness against both the old and new models. If any task category regresses by more than 2 points and the regression is statistically significant (p < 0.05 on a paired bootstrap test), the change needs investigation before deployment.

Putting it together: the CLI

I wrap everything in a CLI that can be run manually or from CI:

# Run eval against a deployed endpoint
python eval_harness.py \
    --endpoint https://api.example.com \
    --model llama-3.1-8b-ft-v3 \
    --suite eval_data/ \
    --baseline results/llama-3.1-8b-ft-v2.json \
    --output results/llama-3.1-8b-ft-v3.json \
    --concurrency 20

# Compare two saved result files
python eval_harness.py compare \
    results/llama-3.1-70b-int4.json \
    results/llama-3.1-8b-ft-v3.json

The results file is JSON with every test case, its score, and the model's raw output. This is critical for debugging: when a regression appears, you want to see exactly which examples the new model gets wrong.

What to evaluate beyond accuracy

Quality is not just correctness. For a deployed model, I also track:

Continuous evaluation

The eval harness should not just run before deployments. Set it up to run continuously against production:

An eval harness is not a one-time project. It is infrastructure. Build it early, run it often, and trust the numbers over vibes. Every model change, every quantization, every prompt tweak should go through the harness.

Next: designing the full inference stack, where we pull together everything from this series into a coherent architecture.