Cut Over the Model Path or Don't Ship: A Fail-Closed Inference Checklist

If production can still reach the model endpoint you used while drafting, you did not cut over. A green deploy does not prove that. It only proves the process started. Treat the inference path as a promotion surface. Pin the base URL, the model identity, the timeout, and the fallback. If any of those still point at a lab or shared drafting host, fail the gate. Do not “try prod first and fall back to the sandbox.” That pattern turns an outage into a data leak. This is not a vibe-coding essay. It is a copy-paste checklist plus a small policy checker you can run in CI. Use it when an AI feature is about to leave the branch. The failure you are actually shipping You added a chat box, a summarizer, or a “explain this diff” button. During development the client pointed at whatever answered quickly. That is normal. It is also how lab hosts become silent production dependencies. The bug is not “we used a model.” The bug is an unnamed path. Someone pasted a base URL into a helper. A generated client baked it into a default. A retry wrapper treats HTTP 429 from prod as a reason to call the drafting host. None of that shows up in a unit test that mocks complete() . You need evidence that prod traffic cannot reach the lab path. You also need evidence that when prod is unhealthy, the feature fails closed instead of wandering. What “cut over” means here Cut over is not “we created a prod API key.” Cut over means all four of these are true at once: - The production base URL is an allowlisted origin, not an environment default inherited from .env.local . - The model identifier is an explicit production contract, not default ,latest , or an empty string. - Timeouts, token ceilings, and retry counts are set in prod config and are smaller than “keep hammering.” - Fallback is deny-by-default: no lab host, no alternate vendor you have not reviewed, no “best effort” second try against the drafting server. If you cannot show those four, you are still in the sandbox. Ship the feature flag off. Six gates, each with a receipt Every gate needs an artifact. A Slack “looks good” is not a receipt. Gate 1 - Named production origin Pass: INFERENCE_BASE_URL in the production secret store matches a committed allowlist. The value is HTTPS, has no path wildcards, and is not a personal tunnel. Fail closed: any prod manifest that still contains localhost , 127.0.0.1 , ngrok , trycloudflare , or a hostname tagged lab , dev , sandbox , or draft . Receipt: the commit SHA of inference-allowlist.json plus the secret-store version id. Gate 2 - Model identity is not a moving alias Pass: production sets INFERENCE_MODEL_ID to a pinned id your vendor or self-hosted gateway documents. The same id appears in the runbook. Fail closed: latest , auto , empty, or a name that only exists on the drafting server. Receipt: a one-line contract in the repo: model id, max tokens, and who owns rotation. Gate 3 - Lab host denylist in CI Pass: CI greps deployable config (Helm, Terraform, Docker Compose prod overlay, sealed secrets templates) and fails on denylisted hosts. Fail closed: denylist bypassed with “temporary” comments, or the check only scans src/ and ignores deploy/ . Receipt: the CI job log URL for the merge commit. Gate 4 - Budgets are numbers, not folklore Pass: production config sets request timeout, connect timeout, max retries, and a per-request token ceiling. Retries do not change the destination. Fail closed: unlimited retries, exponential backoff with no cap, or a retry that swaps base_url . Receipt: the config snippet and a test that asserts the client is constructed once with prod settings. Gate 5 - Fallback cannot re-enter the lab Pass: on timeout, 5xx, or quota errors, the feature returns a user-visible failure and an internal metric. No second client. Fail closed: “if prod fails, call the free server so the demo still works.” Receipt: a failing integration test that stubs prod errors and asserts the lab host is never dialed. Gate 6 - Traffic identity Pass: production requests send a stable X-Service / User-Agent and an environment tag prod . Logs can answer “which app, which model id, which base URL” without reading source. Fail closed: the drafting client and the prod client are the same binary with the same defaults. Receipt: one redacted log line from a staging call that already uses the prod origin. Copy-paste checklist Use this in the PR. Check a box only when the artifact exists. - [ ] inference-allowlist.json lists the single prod origin (or the exact set of regional origins). - [ ] inference-denylist.json lists lab, draft, and tunnel hosts. - [ ] Prod overlay sets INFERENCE_BASE_URL andINFERENCE_MODEL_ID ; neither is blank. - [ ] Client construction test fails if base_url is missing. - [ ] Error-path test fails if any HTTP call target is on the denylist. - [ ] Timeout, retry cap, and token ceiling are set in prod, not only in a README. - [ ] Feature flag default is off until Gate 1-5 receipts are linked. - [ ] Runbook names the owner of model-id rotation. - [ ] Rollback is “flag off,” not “point DNS back at the sandbox.” If a box depends on “we will add it after launch,” the gate failed. Artifact: a policy file CI can reject Commit this as inference-policy.json . Keep it boring. Boring is reviewable. { "allow_base_urls": [ "https://inference.prod.example.internal" ], "deny_host_substrings": [ "localhost", "127.0.0.1", "ngrok", "trycloudflare", "lab.", "sandbox.", "draft.", "dev-inference" ], "require_env": [ "INFERENCE_BASE_URL", "INFERENCE_MODEL_ID", "INFERENCE_TIMEOUT_MS", "INFERENCE_MAX_RETRIES", "INFERENCE_MAX_OUTPUT_TOKENS" ], "max_retries": 1, "forbid_fallback_base_url": true } Replace the allow URL with yours. Do not add the drafting host “just for staging” in the same file that production loads. Staging gets its own overlay. Artifact: a fail-closed checker The script below is a proposal you can run locally and in CI. It does not call any model. It only inspects env and text files you pass as deploy roots. Label it unproven against your repo until you execute it once and keep the log. #!/usr/bin/env python3 """Fail closed if prod config can still reach a lab inference host.""" from future import annotations import json import os import sys from pathlib import Path from urllib.parse import urlparse POLICY = Path("inference-policy.json") SCAN_ROOTS = [Path("deploy"), Path("k8s"), Path("infra"), Path(".")] SCAN_SUFFIXES = {".yml", ".yaml", ".json", ".tf", ".env", ".toml"} def load_policy() -> dict: if not POLICY.exists(): print("FAIL: inference-policy.json missing") sys.exit(2) return json.loads(POLICY.read_text()) def host_of(url: str) -> str: parsed = urlparse(url if "://" in url else f"https://{url}") return (parsed.hostname or "").lower() def denied(host: str, needles: list[str]) -> str | None: for needle in needles: if needle.lower() in host: return needle return None def check_env(policy: dict) -> list[str]: errors = [] for key in policy["require_env"]: if not os.environ.get(key): errors.append(f"missing env {key}") base = os.environ.get("INFERENCE_BASE_URL", "") if base: host = host_of(base) hit = denied(host, policy["deny_host_substrings"]) if hit: errors.append(f"INFERENCE_BASE_URL host matches denylist '{hit}': {host}") allowed_hosts = {host_of(u) for u in policy["allow_base_urls"]} if host not in allowed_hosts: errors.append(f"INFERENCE_BASE_URL host not allowlisted: {host}") model = os.environ.get("INFERENCE_MODEL_ID", "") if model.lower() in {"", "latest", "auto", "default"}: errors.append(f"INFERENCE_MODEL_ID is not pinned: {model!r}") try: retries = int(os.environ.get("INFERENCE_MAX_RETRIES", "99")) except ValueError: retries = 99 errors.append("INFERENCE_MAX_RETRIES is not an int") if retries > int(policy["max_retries"]): errors.append(f"retries {retries} exceed policy max {policy['max_retries']}") return errors def check_files(policy: dict) -> list[str]: errors = [] needles = policy["deny_host_substrings"] for root in SCAN_ROOTS: if not root.exists(): continue for path in root.rglob("*"): if not path.is_file() or path.suffix.lower() not in SCAN_SUFFIXES: continue if path.name == POLICY.name: continue # Skip local-only samples; prod overlays must still pass. if ".local." in path.name or path.name.endswith(".example"): continue text = path.read_text(errors="ignore") lower = text.lower() for needle in needles: if needle.lower() in lower and "prod" in path.parts: errors.append(f"{path}: denylist hit {needle!r}") return errors def main() -> int: policy = load_policy() errors = check_env(policy) + check_files(policy) if errors: print("FAIL-CLOSED: inference cutover incomplete") for item in errors: print(f" - {item}") return 1 print("PASS: inference path looks cut over (config scan only)") return 0 if name == "main": raise SystemExit(main()) Wire it so merge is impossible when it exits 1: # .github/workflows/inference-cutover.yml name: inference-cutover on: pull_request: paths: - "deploy/" - "k8s/" - "infra/**" - "inference-policy.json" - ".github/workflows/inference-cutover.yml" jobs: fail-closed: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Require prod env in the CI job (from secrets, not the PR) env: INFERENCE_BASE_URL: ${{ secrets.INFERENCE_BASE_URL }} INFERENCE_MODEL_ID: ${{ secrets.INFERENCE_MODEL_ID }} INFERENCE_TIMEOUT_MS: ${{ secrets.INFERENCE_TIMEOUT_MS }} INFERENCE_MAX_RETRIES: ${{ secrets.INFERENCE_MAX_RETRIES }} INFERENCE_MAX_OUTPUT_TOKENS: ${{ secrets.INFERENCE_MAX_OUTPUT_TOKENS }} run: python3 scripts/check_inference_cutover.py Secrets belong in the store, not in the PR description. If CI has no prod URL, the job must fail. A skipped check is an open gate. Decision table | Symptom | What you might tell yourself | Fail-closed action | |---|---|---| | Drafting host still in prod overlay | “Staging needs it” | Split overlays. Prod overlay cannot parse the lab hostname. | Model id is latest | “We want improvements automatically” | Pin an id. Rotate with a ticket and a replay te