I Moved My Discord Bot Off a Paid LLM API. Five Things Broke.

My Discord bot has one job: when someone pastes a stack trace into our server's #help channel, it rewrites the trace into a plain-English explanation plus a likely fix. It ran for months on a paid LLM API at a few dollars a month - small money, but the kind that nags you when the workload is trivially bursty: silent for days, then thirty requests in an evening when a game update breaks everyone's mods. So I pointed it at a free endpoint instead. The migration took an afternoon. The consequences of the migration took two weeks to fully shake out, because nothing failed loudly. Everything failed slightly. This is a log of the five things that broke, what each one taught me about provider portability, and the wrapper code that now sits between my bot and any LLM endpoint so I never have to debug these twice. The endpoint I moved to is MonkeyCode, which offers free model access and a free server option - the right price shape for a hobby bot with spiky traffic. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Every failure below, though, is generic to switching OpenAI-compatible providers. I verified that by deliberately reproducing two of them against a second endpoint. Treat this as a portability field report, not a product review. Break 1: The model name was a lie I told myself My config had the model name hardcoded in three places: the request builder, the logging line, and a cost-estimation function that divided tokens by a per-model price. When I swapped providers, I updated one of the three. The bot ran fine - and my logs confidently recorded the old model name for a week, which made every later debugging session confusing because I was reading telemetry for a model I wasn't using. Fix: model identity is configuration, not code, and it flows from exactly one source. # llm_config.py - single source of truth from dataclasses import dataclass import os @dataclass(frozen=True) class LLMConfig: base_url: str api_key: str model: str # whatever the CURRENT provider's catalog calls it max_output_tokens: int timeout_s: float def load_config() -> LLMConfig: return LLMConfig( base_url=os.environ["LLM_BASE_URL"], api_key=os.environ["LLM_API_KEY"], model=os.environ["LLM_MODEL"], max_output_tokens=int(os.environ.get("LLM_MAX_OUTPUT", "600")), timeout_s=float(os.environ.get("LLM_TIMEOUT_S", "20")), ) Boring, obvious, and the root cause of the messiest week. A related trap: never assume a model name means the same thing across providers, or even across months on the same provider. Catalogs change. Check what your endpoint actually serves before writing it into an environment variable, and expect to change it again. Break 2: Rate limits arrived as success-shaped failures On the paid API, I had never once seen a 429. On a free tier, the evening burst pattern hit the limit within the first weekend. The truly bad part: my error handler treated non-200 responses as "log and move on," so during peak hours the bot silently ignored half the stack traces posted. Users thought it was ignoring them specifically and reposted, which made the burst worse. A rate limit had become a feedback loop. Fix: bounded retries with jittered backoff, and - critically - a visible degraded state. # llm_call.py - retry wrapper with honest degradation import asyncio import random import openai MAX_ATTEMPTS = 4 async def call_with_backoff(client, cfg, messages) -> str | None: for attempt in range(MAX_ATTEMPTS): try: resp = await asyncio.to_thread( client.chat.completions.create, model=cfg.model, messages=messages, temperature=0.2, max_tokens=cfg.max_output_tokens, timeout=cfg.timeout_s, ) return resp.choices[0].message.content except openai.RateLimitError: if attempt == MAX_ATTEMPTS - 1: return None # caller must say so out loud base = min(2 ** attempt, 8) await asyncio.sleep(base + random.uniform(0, base)) except openai.APITimeoutError: return None # a slow answer in a chat channel is a wrong answer return None async def explain_trace(client, cfg, trace: str) -> str: result = await call_with_backoff(client, cfg, [ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": trace}, ]) if result is None: return ("⚠️ I'm rate-limited right now. Try again in a minute, " "or trim the trace to the first error line - shorter " "messages are more likely to get through.") return result The degraded message does two jobs: it resets user expectations, and it gives them an actionable workaround (shorter input). Silence is the worst possible response to a rate limit in an interactive product. Break 3: Context length differences truncated my prompts mid-word The old setup had headroom I never measured, so my prompt builder happily concatenated the system prompt, the full stack trace, and the last five channel messages for context. On the new model, long traces started returning explanations of the wrong error - because the actual exception line, at the end of the paste, had been truncated away by the context window. No exception was raised. The model just answered a different question than the user asked. This is the nastiest class of provider-migration bug: output stays plausible while becoming wrong. Fix: budget tokens explicitly and truncate deliberately, keeping the semantically important part. For stack traces, the important part is the last exception block, not the first frames: def trim_trace(trace: str, char_budget: int) -> str: """Keep the tail of the trace: the exception line matters most.""" if len(trace) dict: start = asyncio.get_event_loop().time() result = await call_with_backoff(client, cfg, [ {"role": "user", "content": "Reply with exactly: PROBE_OK"}, ]) elapsed = asyncio.get_event_loop().time() - start return { "ok": result is not None and "PROBE_OK" in result, "latency_s": round(elapsed, 2), } Two weeks of probe data also gave me something I never had on the paid API: an honest picture of the free tier's behavior under my real traffic pattern - when bursts collide with limits, what p50 latency actually feels like. If you run one thing from this article, run the probe. It converts "the free tier seems flaky" into data you can act on. What I'd tell anyone doing the same migration The switch was worth it: the bot's operating cost for this workload went to zero, and every failure above was a latent bug in my code that the paid tier's headroom had been papering over. Free capacity is unforgiving in a useful way. But be honest about the fit: - Good candidates: bursty hobby tools, bots, personal automations, internal utilities - workloads where a retry and a polite degraded message are acceptable, and where request volume would otherwise generate a real bill. - Bad candidates: anything with a latency SLA, user-facing flows where a failed call breaks a purchase or signup, and workloads with steady high volume that will live permanently against rate limits. If your traffic graph is a flat line, a free tier is not your production plan; it's your development environment. - Either way: build the portability wrapper first - single-source config, backoff with visible degradation, deliberate truncation, a synthetic probe - and the choice of provider becomes a config change you can revisit quarterly instead of a commitment. If you want a concrete place to try this, MonkeyCode's free model access and free server option are what this bot currently runs on; the wrapper above means it could run somewhere else tomorrow, which is precisely the property worth engineering for. The migration took an afternoon. The engineering to make the next migration boring took two weeks - and it's the part that was actually worth doing. Top comments (0)