The name comes from the canaries miners carried to warn of bad air. A bank switching its support assistant to a new model routes 2% of conversations to the new version for a day. Dashboards compare that 2% with the other 98%: error rates, latency, cost per conversation, escalations to humans, and a sampled quality score from LLM-as-a-judge. If the canary looks no worse, traffic steps up to 10%, then 50%, then everyone. If a metric crosses a threshold, traffic goes back to the old version automatically.
The goal is to limit the damage of a bad release, not to prove the new version is better. That makes it different from an A/B test, which is designed to measure improvement, though Google’s SRE guidance notes that canarying is effectively an A/B process and uses the same comparison. A shadow deployment goes one step safer: the new version sees real traffic but its outputs never reach users.
AI releases are harder to canary than ordinary code. Quality problems often don’t raise errors; a model can answer every request politely and still be wrong more often. The slice may need to run longer and include enough examples to judge quality, and the comparison should include eval scores, not just uptime. Keep the previous model or prompt version ready to restore, which is easier with a model registry and prompt management. See also MLOps and LLMOps.
Sources
- Google SRE Workbook: Canarying Releases — Defines canarying as a partial, time-limited deployment of a change and its evaluation, with the canary compared against a control, and says canarying is effectively an A/B testing process; checked October 24, 2026.
- Martin Fowler (Danilo Sato): Canary Release — Describes rolling a change out to a small subset of users before the whole infrastructure to reduce the risk of a new software version; checked October 24, 2026.