Silent Model Fallback: A Five-Check Debug Workflow for Agent Jobs

Green jobs with wrong output are usually envelope bugs, not prompt bugs. Log the response envelope before you touch a single prompt token.

This post walks through a debugging workflow. The scenario below is explicitly illustrative, not a reproduced incident, and the code is unexecuted sample code you should adapt.

The illustrative symptom

A nightly batch job writes 50,000 structured records. One morning it finishes with exit code 0. Records parse. Fields look plausible but subtly off: shorter lists, softer wording, missing edge cases. Nothing threw. Nothing retried. That is the hard part.

Three families of cause dominate here:

• A different model served the request than the one you asked for.
• Your prompt assembly changed without a version bump.
• Decoding parameters drifted between environments.

All three are invisible if you only log the text.

Step 1: Capture the envelope, not just the text

Log the request identity and the served identity together. This is the single highest-value change.

# illustrative only; adapt to your provider SDK
import hashlib, json, logging, time
log = logging.getLogger("llm")

def call(client, requested_model, messages, **params):
    payload = {"model": requested_model, "messages": messages, **params}
    started = time.time()
    resp = client.chat.completions.create(**payload)
    record = {
        "requested_model": requested_model,
        "served_model": getattr(resp, "model", None),
        "finish_reason": resp.choices[0].finish_reason,
        "usage": resp.usage.model_dump() if resp.usage else None,
        "prompt_sha256": hashlib.sha256(
            json.dumps(messages, sort_keys=True).encode()
        ).hexdigest()[:16],
        "params": {k: v for k, v in params.items() if k != "messages"},
        "latency_ms": int((time.time() - started) * 1000),
    }
    log.info("llm_call %s", json.dumps(record))
    return resp

served_model may legitimately differ from requested_model. Providers alias versions and route traffic. If you never compare them, you never notice.

Step 2: Query your logs for drift

One command answers the first question: did the model change?

# assumes one JSON object per line in llm.jsonl
jq -r 'select(.served_model != .requested_model) | [.requested_model, .served_model] | @tsv' llm.jsonl \
  | sort | uniq -c | sort -rn

Then check prompt stability across a run:

jq -r '.prompt_sha256' llm.jsonl | sort | uniq -c | sort -rn | head

A handful of hashes is expected. Hundreds of hashes for identical inputs means something is injecting variability, such as timestamps, unordered dicts, or retrieved context.

Step 3: Turn silent drift into a hard error

Fail fast when the served model is not in your approved set. Fill the set from the provider's current documentation, not from memory.

# illustrative allowlist; populate from provider docs you have verified
APPROVED = {"<served-model-id-from-docs>"}

def assert_served(resp, expected):
    served = getattr(resp, "model", None)
    if served not in APPROVED or served != expected:
        raise RuntimeError(f"model drift: requested={expected} served={served}")

If strict failure is too disruptive, tag records instead. Write model_id into every output row. Wrong rows become traceable, not permanent mysteries.

Step 4: Add a canary set with property assertions

Do not assert exact strings. Assert properties that must hold regardless of wording.

Run the canary before the batch, not after. Ten fixed inputs with known invariants is enough to catch most envelope bugs.

Step 5: Separate retries from fallbacks

Retries and fallbacks are different policies. Mixing them hides failures.

• Retry only on transport errors, rate limits, and timeouts.
• Never retry on schema failure. That hides the bug and doubles cost.
• A fallback to another model is a product decision. Tag it and surface it.

# illustrative policy sketch
RETRYABLE = {"429", "500", "502", "503", "504"}

def classify(err):
    code = getattr(err, "status_code", None)
    return "retry" if str(code) in RETRYABLE else "fail_loud"

Where a free tier and a free server fit

This harness needs a sandbox: run the canary, log envelopes, break something, repeat. That is a better fit for free capacity than production traffic.

The MonkeyCode project is open source and, per the operator, offers free model access and a free server option. Token allowance, model lineup, hardware, and duration are operational details that change. Verify current figures on the project's own page before you size anything against them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A practical division of labour for the workflow above:

Limitations and who should skip this

The envelope log costs storage. At high volume, sample it or keep only hashes. This workflow will not save you if the real defect sits in your schema validator. It also will not help with nondeterministic decoding you actually want.

Skip this approach if you need contractual model pinning, per-request SLAs, or audit trails tied to a specific published version. Those require vendor guarantees, not a debugging method.

A short checklist

Before blaming the model, confirm five things:

• requested_model and served_model appear in every log line.
• A drift query runs over the last full batch.
• One prompt hash exists per logical input.
• Canary assertions cover schema, enums, lengths, and finish reasons.
• Fallback is a labelled decision, never a silent retry.

If you want a place to try step 4 without burning production quota, the project's free model access and free server option are a reasonable starting sandbox. Check the current terms yourself before relying on them.