Your cheap LLM relay might be swapping the model. Here's how to catch it.

If you buy model access through a cheap "GPT / Claude / DeepSeek" API relay, you have a trust problem nobody puts on the pricing page: some relays quietly swap in a smaller, quantized, or context-truncated model and bill you for the flagship. The nasty part is the timing - it often works fine on day one and degrades weeks later, right after you've stopped checking.

This article is the verification playbook I wish existed when I started evaluating gateways. It works against any OpenAI- or Anthropic-compatible endpoint, and at the end there's a small open-source tool that automates the whole thing.

Disclosure up front: I work on daoxe, one such gateway. I'm writing the verification method so you can test any provider - including us. A verification tool is only useful if it will just as happily flag the people who wrote it.

The one check that doesn't work

The obvious move is to ask the endpoint: "What model are you?" Don't bother. That answer comes from a system prompt or a fine-tune and is trivially spoofable in both directions. A relay serving you a 7B model can print "I am GPT-4-class" all day; a genuine flagship can be told to call itself something else. Self-report is theatre.

You have to test capability and behavior, not what the model claims about itself. And you have to measure - one request tells you nothing, because temperature, sampling, and routing all add noise.

The four cheats you're actually guarding against

When margins are thin, a dishonest relay can:

• Substitute the model. You ask for a flagship; they serve a cheaper one and bill you for the flagship.
• Serve heavy quantization. Same model name, aggressively quantized weights, visibly worse reasoning.
• Truncate your context. They silently cap your context window to save tokens, so long-context tasks quietly fail.
• Fall back without telling you. When the upstream is rate-limited, they route you to whatever's available and never surface it.

Each of these leaves a behavioral fingerprint. Here's how to read them.

Five signals that actually catch it

None of these individually proves anything. Together, they either reassure you or give you a reason to look closer.

1. Tokenizer fingerprint

Different model families use different tokenizers, and the tokenizer leaks through the usage.prompt_tokens the server reports. Send a battery of crafted strings - digit runs, whitespace, CJK, emoji, code, URLs, mixed unicode - and record how many tokens each costs. To cancel the fixed per-request chat-template overhead, measure a delta: tokens(anchor + probe) − tokens(anchor).

• Best case (reference-free): diff that token vector against an endpoint you trust serving the same model id. A different vector means a different tokenizer, which means a different model family - a high-confidence signal.
• Single endpoint: classify the vector against known tokenizer tables (e.g. OpenAI's cl100k_base / o200k_base). If the claimed family has no public reference (Claude, Gemini…), the honest answer is inconclusive - use a diff, not a guess.

2. Capability floor

A handful of tasks with objectively checkable answers you already know: multi-step arithmetic, string reversal, character counting, plus a strict-JSON format test. These are trivial for any full-tier model; a downgraded or heavily quantized substitute is more likely to trip. Weight failure more than success - passing an easy task proves little, but a "flagship" that fails a floor task is worth surfacing.

3. Long-context recall (the needle)

Generate deterministic filler text, hide a unique passphrase in the middle, and ask the endpoint to read it back - across several context lengths. Because you placed the needle, you always know the right answer (no external key needed). Recall that works at short context but breaks at longer context is the classic signature of silent context truncation.

4. Stability & performance

Repeat one fixed prompt N times at temperature=0 and look at output determinism, the stability of the server-reported model field and system_fingerprint, plus p50/p95 latency and error rate. A server that reports different model values or wildly different outputs for identical calls may be routing you across different backends.

5. Self-report (low weight, on purpose)

Ask the model to name itself and compare to the claim - but cap its confidence at low, because it's spoofable both ways. Include it only because its absence would be conspicuous; never let it drive a verdict.

The design rules that make the signals trustworthy

• temperature=0, fixed max_tokens, fixed system prompt - remove randomness.
• Measure, don't eyeball - repeat and record percentiles.
• Diff against a reference you trust whenever you can. This is the single strongest move.
• Re-run on a schedule. Silent degradation is a time-series problem, not a launch-day one.

Do it by hand (the 60-second version)

You can start without any tooling. Pin the parameters and fire the same probe at two endpoints:

# Your relay
curl -s -w '\n%{time_total}s\n' https://your-relay.example/v1/chat/completions \
  -H "Authorization: Bearer $RELAY_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"MODEL_ID","temperature":0,"max_tokens":64,
       "messages":[{"role":"user","content":"Reverse this exactly: abcdefghij. Then count the letters."}]}'

# The official API, same model, as your reference
curl -s -w '\n%{time_total}s\n' https://api.openai.com/v1/chat/completions \
  -H "Authorization: Bearer $OFFICIAL_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"MODEL_ID","temperature":0,"max_tokens":64,
       "messages":[{"role":"user","content":"Reverse this exactly: abcdefghij. Then count the letters."}]}'

Watch the usage.prompt_tokens, the answer quality, and the latency. Do it a few times. Divergence is your first thread to pull.

Automate it: an open-source probe

Running that battery by hand every week is tedious, so there's a small open-source CLI, llm-honesty-probe, that does it for you. Design goals worth calling out:

• Zero runtime dependencies. Pure Python standard library - nothing to pip install, nothing in a lockfile to audit. Clone it and read it before you run it.
• Your key is never printed, logged, or stored. It's read only from an environment variable (there's deliberately no --key flag, so it can't land in shell history), and every line of output passes through a redaction layer.
• Provider-neutral. No allow-list, no "preferred" endpoint, no telemetry. It treats the official API and any relay identically.

git clone https://github.com/seven7763/llm-honesty-probe
cd llm-honesty-probe
python3 -m llm_honesty_probe --self-test
# runs against a built-in mock; no key needed

# Single endpoint: does it behave like the model it claims?
export OPENAI_API_KEY="sk-...your key..."
# never appears in argv or history
python3 -m llm_honesty_probe \
  --base-url https://any-provider.example/v1 --claimed-model MODEL_ID

# Differential (strongest): diff the relay against the official API
export OPENAI_API_KEY_OFFICIAL="sk-...official..."
python3 -m llm_honesty_probe \
  --base-url https://cheap-relay.example/v1 --claimed-model MODEL_ID \
  --compare-base-url https://api.openai.com/v1 --compare-model MODEL_ID \
  --compare-api-key-env OPENAI_API_KEY_OFFICIAL

# Claude / Anthropic-shaped endpoints:
export ANTHROPIC_API_KEY="sk-ant-..."
python3 -m llm_honesty_probe \
  --base-url https://any-provider.example --protocol anthropic --claimed-model claude-sonnet-...

The output marks each signal [OK] (consistent), [!!] (suspicious), or [--] (inconclusive), each with a confidence level. Use --json --out report.json to diff results in cron/CI over time - which is the whole point, since degradation shows up as drift, not on launch day.

Read the limitations, seriously

Overstating an honesty tool defeats its purpose. So, plainly:

• These are signals, not proof. Nothing here cryptographically establishes which weights a provider runs. A provider that mirrors the tokenizer and matches capability and holds context and stays stable is, for practical purposes, giving you the model - but you can't prove intent from the outside.
• The probe set is small and known. A provider that special-cases these exact probes could game a single-endpoint run. That's why the strongest mode is a differential diff you control.
• "Suspicious" has innocent explanations. A smaller model can be the correct answer if that's what you asked for; a short context limit can be the model's real limit; different infra changes latency. Use --compare to disambiguate.
• temperature=0 isn't perfectly deterministic on real hardware, so the determinism signal is intentionally low confidence.

Treat a clean run as reassuring, not a guarantee. Treat a flag as a reason to look closer, not a conviction.

What "honest" looks like from a provider

The reason I can publish this without flinching is that verifiability is the whole pitch. A gateway worth using should be:

• Benchmarkable. Fine with you running this against it. (If a provider gets cagey when you ask to benchmark, that's your answer.)
• Native on the protocols you use. OpenAI Chat Completions and Anthropic Messages, so Claude Code works properly - not just the OpenAI shape.
• One key, many models, with an account-scoped /v1/models so what you see is what you can call.
• Transparent on cost and status - itemized usage, nothing about billing hidden.

daoxe is built around exactly that: an OpenAI-compatible base URL (https://daoxe.com/v1), native Anthropic Messages for Claude Code, one key across many models, and a design meant to be verified rather than trusted. It's not available in mainland China.

Please don't take my word for any of it - point the probe at it and at whatever you use today, and compare. If daoxe ever fails these checks, that's a bug report we want.

Complementary tool: llm-gateway-benchmark

llm-gateway-benchmark answers "is this endpoint fast and cheap?" (success rate, p50/p95, $/1M tokens). This one answers "is it actually the model it claims?" They're complementary, not competing.

TL;DR

• Don't ask the model what it is. Do test capability and behavior.
• Five signals: tokenizer fingerprint, capability floor, long-context needle, stability, and (low-weight) self-report.
• Pin temperature=0, measure percentiles, diff against a reference, and re-run weekly.
• Signals, not proof - but signals are exactly what you're missing right now.