Software CI/CD has a beautiful, boring shape: commit, test, build, deploy, rollback. Inference CI/CD has the same shape but every step is heavier. The artifact is a model, not a binary. The test is a latency budget, not a unit test. And the rollback can mean swapping a GPU.
Here's what I've learned running this for real.
The artifact is the problem
In normal CI, your artifact is a few MB. In inference, your artifact is a model: 8GB for a small one, 140GB for a 70B in FP16. You can't store it in a registry like a Docker image and pull it in seconds.
The answer is a model registry backed by object storage (S3, GCS), with the model as the immutable artifact. The registry tracks metadata: base model, quantization, precision, eval scores, who built it, when. The deployment is a pointer to a registry entry, not the weights themselves.
The test is a latency budget
You can't unit test a model. You can't assert "this model is correct". What you can do is run a benchmark suite: a fixed set of prompts, measured for TTFT, TBT, and quality. The gate is a latency budget, not a pass/fail.
My pipeline runs three gates:
- Smoke: does the model load and serve one request? Fast, catches the obvious.
- Bench: TTFT/TBT p50/p95 on a standard prompt set. Catches regressions from quantization or engine changes.
- Eval: quality on a held-out set. Catches the silent regressions that latency tests miss.
Each gate is a real measurement on real hardware, not a mock. The bench gate is the one that saves the weekend: it catches the "quantization made it 2x faster but 30% dumber" problem before it ships.
The deploy is a rollout, not a switch
When the model passes, you don't flip a switch. You do a canary rollout: route 5% of traffic to the new model, watch the latency and quality dashboards, then ramp. If TTFT p95 spikes, you roll back to the previous registry entry, not to a previous commit.
The rollback is the part people forget. With code, rollback is a git revert. With inference, rollback is re-pointing the deployment at the old model and letting the new one drain. That's why the registry matters: it makes rollback a pointer change, not a re-upload.
Code CI/CD ships behavior. Inference CI/CD ships capability. The pipeline is the same shape, but the artifact is heavier, the test is a measurement, and the rollback is a pointer.
The pipeline, end to end
Every step is automated except the judgment calls, and there are fewer of those than you'd think.
The takeaway
Inference CI/CD is code CI/CD with the weights on. Same discipline, heavier artifacts, and a latency budget where the unit test used to be.