Workshop: Catch Shared-Host Drift With a Five-Probe Floor Card in 85 Minutes

Shared inference hosts fail quietly when a probe set is missing, not when a dashboard still looks green. A five-probe floor card records the cheapest behaviors you still require after a host, prompt, or routing change. This workshop times that card at eighty-five minutes, with rerunnable files and a pass-or-fail ledger. Students leave with a JSON pack, a local scorer, and a printed floor they can recheck after every lab swap. Why a floor card beats a chat anecdote Anecdotes from one lucky prompt hide regressions that only appear on the second or third frozen case. Shared hosts also move under you, because capacity, routing, and hidden system prompts are not a public version pin. You need a frozen input set, a deterministic scorer, and a stored floor, or next week's run cannot be compared at all. The method below treats the model as an untrusted function with a tiny, documented surface. Measurement talk in developer circles often outruns the tests that still discriminate. When a host starts returning fluent prose, empty objects, or a constant label, yesterday's demo stops being evidence. A floor card is deliberately small so a classroom can finish it, not so a vendor can be ranked. If the card cannot fail, it cannot teach anything useful about drift. What this workshop is not This is not a leaderboard, a latency study, or a claim about production model quality on a named system. It is a teaching lab that detects silent capability loss on one narrow, field-shaped task. If your team already runs a versioned eval platform with owners and an SLA, skip this outline and keep that platform. If you cannot freeze inputs because the task is open-ended prose, shrink the contract before you schedule the lab. Workshop clock Keep a visible timer for eighty-five minutes and refuse to expand the probe set during the first pass through the files. - 0-10 min - Frame the failure. Name one task, one JSON contract, and one host URL environment variable. - 10-25 min - Write the probe pack. Freeze five inputs, expected fields, and scoring modes on disk. - 25-45 min - Run the worked example. Score against a file fixture before any remote host is allowed. - 45-65 min - Exercise: point at a shared host. Keep the scorer identical and record a second floor card. - 65-80 min - Exercise: break one probe. Mutate a prompt or response shape and show the ledger turning red. - 80-85 min - Recap limits. List who should not ship this card as a quality gate. Artifact layout Students should create a directory they can zip, copy to another machine, and rerun without editing expects. probe-floor/ probes.json score_probes.py floor_card.json Makefile The Makefile is the only class entrypoint, which stops ad-hoc flags from becoming the real curriculum. .PHONY: fixture remote diff fixture: python3 score_probes.py --probes probes.json --base-url file://fixtures --out floor_card.json remote: python3 score_probes.py --probes probes.json --base-url "$$PROBE_BASE_URL" --out floor_card.remote.json diff: python3 score_probes.py --diff floor_card.json floor_card.remote.json Probe pack schema Keep scoring modes boring on purpose. Exact field match plus JSON parse success catch more host drift than a long prose rubric. { "task": "semver_bump_from_diff", "contract": { "type": "object", "required": ["bump", "reason"], "properties": { "bump": {"enum": ["major", "minor", "patch", "none"]}, "reason": {"type": "string", "minLength": 8, "maxLength": 160} } }, "floor": {"min_pass": 5, "max_parse_fail": 0}, "probes": [ { "id": "P1_docs_patch", "input": {"diff_summary": "docs: fix typo in README install block"}, "expect": {"bump": "patch"} }, { "id": "P2_optional_field_minor", "input": {"diff_summary": "feat: add optional timeout_ms to ClientConfig"}, "expect": {"bump": "minor"} }, { "id": "P3_removed_field_major", "input": {"diff_summary": "breaking: remove ClientConfig.retry_count"}, "expect": {"bump": "major"} }, { "id": "P4_empty_diff_none", "input": {"diff_summary": ""}, "expect": {"bump": "none"} }, { "id": "P5_chore_none", "input": {"diff_summary": "chore: reformat imports with no behavior change"}, "expect": {"bump": "none"} } ] } Five probes are a floor, not coverage, and they exist to fail closed when a host collapses. Watch for fluent prose, empty JSON, or a constant minor returned for every distinct case. If a pair wants a sixth probe during the first hour, park it in a notes file instead of changing the pack. Worked example students can rerun Label: this example is a local teaching fixture, not a measured vendor benchmark and not a claim about any hosted model. The file backend returns canned JSON so the scorer can be graded without a network round trip. Remote calls below use a lab default path; change that path to match whatever route your host actually documents. # score_probes.py - teaching example, not a production eval platform from future import annotations import argparse, json, sys, urllib.request from pathlib import Path SYSTEM = ( "Return only JSON with keys bump and reason. " "bump must be major, minor, patch, or none." ) def load_probes(path: Path) -> dict: return json.loads(path.read_text()) def complete_file(probe: dict) -> str: bump = probe["expect"]["bump"] return json.dumps({"bump": bump, "reason": f"fixture:{probe['id']}"}) def complete_http(base: str, probe: dict, timeout: float = 30.0) -> str: payload = json.dumps({ "messages": [ {"role": "system", "content": SYSTEM}, {"role": "user", "content": json.dumps(probe["input"])}, ] }).encode() req = urllib.request.Request( base.rstrip("/") + "/v1/chat/completions", data=payload, headers={"Content-Type": "application/json"}, method="POST", ) with urllib.request.urlopen(req, timeout=timeout) as resp: body = json.loads(resp.read().decode()) return body["choices"][0]["message"]["content"] def score_one(contract: dict, probe: dict, raw: str) -> dict: row = {"id": probe["id"], "pass": False, "parse_ok": False, "detail": ""} try: data = json.loads(raw) except json.JSONDecodeError: row["detail"] = "not_json" return row row["parse_ok"] = True if set(contract["required"]) - set(data): row["detail"] = "missing_keys" return row if data.get("bump") != probe["expect"]["bump"]: row["detail"] = f"bump:{data.get('bump')}" return row reason = data.get("reason", "") if not isinstance(reason, str) or not (8 dict: rows = [] for probe in probes["probes"]: raw = ( complete_file(probe) if base_url.startswith("file:") else complete_http(base_url, probe) ) rows.append(score_one(probes["contract"], probe, raw)) passed = sum(1 for r in rows if r["pass"]) parse_fail = sum(1 for r in rows if not r["parse_ok"]) floor = probes["floor"] return { "task": probes["task"], "passed": passed, "parse_fail": parse_fail, "floor_ok": passed >= floor["min_pass"] and parse_fail int: print(f"local_floor_ok={a['floor_ok']} remote_floor_ok={b['floor_ok']}") ids = {r["id"]: r for r in a["rows"]} rc = 0 for row in b["rows"]: prior = ids.get(row["id"], {}) if prior.get("pass") and not row["pass"]: print(f"REGRESS {row['id']} {prior.get('detail')} -> {row['detail']}") rc = 1 elif prior.get("pass") != row["pass"]: print(f"CHANGE {row['id']} pass {prior.get('pass')} -> {row['pass']}") rc = 1 return rc def main() -> int: p = argparse.ArgumentParser() p.add_argument("--probes") p.add_argument("--base-url") p.add_argument("--out") p.add_argument("--diff", nargs=2) args = p.parse_args() if args.diff: a = json.loads(Path(args.diff[0]).read_text()) b = json.loads(Path(args.diff[1]).read_text()) return diff_cards(a, b) pack = load_probes(Path(args.probes)) card = run(pack, args.base_url) Path(args.out).write_text(json.dumps(card, indent=2) + "\n") print(json.dumps({"floor_ok": card["floor_ok"], "passed": card["passed"]}, indent=2)) return 0 if card["floor_ok"] else 2 if name == "main": sys.exit(main()) Expected fixture command for every pair, before anyone exports a remote URL: python3 score_probes.py --probes probes.json --base-url file://fixtures --out floor_card.json The teaching fixture always returns the expected bump, so floor_ok must be true before PROBE_BASE_URL is set. That order is the lab, not a ceremony around the lab. Exercise 1 - freeze the contract, not the essay (15 minutes) Students often want a long rubric because it feels more serious than five enum checks. Stop that impulse and ask each pair to delete any probe whose expect field is a free-text essay. A probe that cannot fail in one sentence is not frozen yet, and it will be edited to match a nicer model next week. Checklist for the teaching assistant: - Every probe has a stable id that will survive later wording edits. - bump is an enum, never a list of allowed synonyms in natural language. - reason is length-bounded so empty strings and novels both fail the same way. - min_pass equals the probe count on day one; lowering it requires a written note. Exercise 2 - change the host, not the scorer (20 minutes) Export one URL and keep probes.json byte-identical. A local scorer should not care which process sits behind a compatible HTTP path, only whether the floor still holds. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project with free model access and a free server option that can sit behind PROBE_BASE_URL once the file backend already prints floor_ok: true . export PROBE_BASE_URL="http://127.0.0.1:8080" make remote make diff Interpret the diff with a table, not with a vibe from a single chat window in another tab. | Diff signal | Meaning in this lab | Next action | |---|---|---| floor_ok stays true | Floor still holds on this host | Record the card; do not add probes yet | | parse failures greater than zero | Contract broke into prose or invalid JSON | Fix the prompt or reject the host | | one bump mismatch | Task policy drifted on a frozen case | Keep the probe; do not edit expect | all bumps become minor | Classifier collapsed to a constant | Fail the floor; do not average scores | Do not invent a latency num