GPU cost model: dollars per token

Every inference decision is secretly a cost decision. This is the model I use to make them, and it's surprisingly simple: dollars per token.

Here's the math. One H100 costs about $3/hour on demand (or half that reserved). What can it do in an hour?

tokens/hour = throughput(tokens/sec) × 3600
cost per token = $3 / tokens_per_hour

If a 7B model does 200 tokens/sec per replica, that's 720,000 tokens/hour, so:

cost per token = 3 / 720000 ≈ $0.0000042 ≈ $4.2 per million tokens

Compare to API pricing, which for a comparable model is $0.15-$0.60 per million input tokens and $0.60-$2.50 per million output. The GPU is 10-100x cheaper if you keep it busy. That "if" is the whole game.

When self-hosting wins

• High volume. Once you're serving enough tokens, the fixed cost of the GPU amortizes fast.
• Predictable traffic. If you have a baseline load, you can reserve capacity and the per-token cost drops further.
• Latency control. You can pin the model, the batch, the engine, and the routing. No shared-tenant surprises.

When it loses

• Low or spiky volume. If your traffic is 90% idle, you're paying for the GPU while it does nothing. The API is cheaper because you only pay for tokens.
• You can't fill a GPU. A single replica serving 5 RPS is a very expensive way to serve 5 RPS.
• Engineering time. Operating GPUs is a job. If you don't have the team, the API's markup is worth it.

The hidden costs

Per-token API pricing hides the real costs. Self-hosting exposes them:

• KV cache memory means long conversations use more GPU per request, effectively raising cost per token.
• Cold starts mean you keep warm replicas that serve nothing.
• Fragmentation means a GPU at 60% utilization is still a full GPU bill.
• Multi-cloud means you're paying different prices for the same work, and the routing has to know it.

This is why the roofline matters in a meeting, not just in a notebook: every point of arithmetic intensity is a point on the cost curve.

The takeaway

Self-hosting isn't cheaper. Self-hosting at high utilization is cheaper. Everything else is a very expensive way to learn that lesson.

Next: blue-green deployment, because once you've decided to run your own GPUs, you need to ship changes without downtime.