Canary and Blue-Green Deploys for Model Changes

A code deploy either works or throws. A model deploy usually does neither: the new version returns 200s at the same latency and is slightly worse at the thing you care about. That is why model rollouts need a quality gate as well as a health gate, and why the honest question about a canary is not “did it error?” but “have I seen enough requests to tell?” Why model deploys are not code deploys Three properties break the usual assumptions. - Failure is silent and distributional. A 2% drop in extraction accuracy produces no errors, no latency change and no alert. It shows up as support tickets three weeks later. - Output is non-deterministic. You cannot diff two responses and conclude anything from one pair. Comparison has to be statistical, over a sample. - Capacity is the constraint on the strategy. Running blue and green simultaneously means holding two full sets of GPUs. For a large model that doubles the most expensive line in the budget for the duration of the rollout, which is a real reason to prefer a small canary over blue-green. Add one more, which applies when you are switching between hosted models rather than your own weights: providers update models under a stable name. Silent model updates covers that case, and the defence is the same eval gate applied on a schedule rather than on a deploy. Blue-green, canary, and shadow | Strategy | Description | |---|---| | Blue-green | Two complete environments; flip all traffic at once, flip back to roll back. Fastest rollback available and the simplest to reason about. Costs double capacity during the overlap, and gives you no gradual signal - the first evidence of a problem is 100% of users having it. | | Canary | A small fraction of traffic to the new version, increased in steps as metrics hold. Cheap in capacity, and limits the blast radius to the fraction you chose. The cost is time: a 1% canary needs a hundred times as long to accumulate the same evidence. | | Shadow | Real traffic is duplicated to the new version and its responses are recorded and discarded. Zero user risk, and the only shape that lets you compare two answers to the same input. Costs full inference on every shadowed request, and cannot see anything downstream of the response. | They compose, and the sensible default is to compose them: shadow first to catch crashes and gross regressions on real inputs, then canary to catch what shadowing cannot see, then promote. Shadow traffic for LLMs covers the duplication mechanics, including the trap of shadowing requests that have side effects. Splitting traffic deterministically The split must be sticky per user, not per request. A user whose conversation alternates between two model versions gets an inconsistent assistant, and any metric you compute over the session is contaminated. Hash a stable identifier into a bucket: import hashlib def variant(user_id: str, canary_percent: int, salt: str = "chat-model-2026-08") -> str: """Stable assignment: the same user always lands in the same bucket for a given salt. Change the salt to re-randomise a later experiment.""" h = hashlib.sha256(f"{salt}:{user_id}".encode()).digest() bucket = int.from_bytes(h[:4], "big") % 100 # 0..99 return "canary" if bucket " value: 0.5% for: 5m - metric: p95_latency_ms op: ">" value: 1.25 relative_to: stable for: 10m - metric: schema_valid_rate op: " " value: 1.15 relative_to: stable for: 1h - metric: thumbs_down_rate op: ">" value: 1.30 relative_to: stable for: 4h on_abort: - set canary traffic to 0 # seconds; the stable pods never went away - keep canary pods running # for diagnosis, not for traffic - page the on-call, do not auto-promote again without a human The last two lines are the ones people leave out. Keeping the canary pods running after an abort preserves the evidence - logs, traces, a live process to inspect - and setting traffic to zero rather than deleting the deployment makes the rollback take seconds rather than a scheduling cycle. What to do next is in the on-call runbook, which has a page for exactly this symptom. Top comments (0)