LLM Provider Returns 529/503 Overloaded: LiteLLM vs OpenRouter vs a Hand-Rolled Fallback

If one provider's outage takes your feature down, you need a fallback path, and you have three realistic options. A hosted router such as OpenRouter is the least code but adds a third party to your request path.

A self-hosted layer such as LiteLLM keeps keys and logs in your hands but is something you now operate. A 40-line wrapper around two SDKs is enough when you only ever call two models.

The symptom that sends people searching for this looks like 529 overloaded_error, 503 Service Unavailable, or a stream that opens and then dies halfway. I'll walk through what to fix first, then compare the three approaches.

Model names below are placeholders; substitute whatever you run as of November 2026.

Why does retrying the same provider not fix a 529 or 503?

Retries handle blips. Overload and regional incidents last minutes to hours, and a retry loop with backoff just queues your users behind the problem. Worse, if every worker retries, you add load to a provider that is already shedding it.

The dead end I see most often: bumping max_retries in the SDK from 2 to 10. That turns a fast failure into a 90-second hang, and your own request timeouts then fire first.

Retries are for 429s with a Retry-After header and for transient network errors. A different provider is the answer to "this one is down."

Takeaway: retry for blips, fail over for outages, and never use the same mechanism for both.

What should trigger a failover, and what should not?

Before choosing a tool, decide the policy, because every option below is only as good as the rules you feed it.

The row people get wrong is context length. A fallback model with a smaller context window will fail your longest requests, which are usually your most important ones. Check the window of every model in the chain before you ship it.

Takeaway: a fallback chain that ignores 4xx semantics will mask your own bugs as provider flakiness.

How does LiteLLM handle fallbacks?

LiteLLM is an open-source Python library plus an optional proxy server that exposes an OpenAI-compatible API in front of many providers. Its Router takes a list of deployments and a fallback map.

import os
from litellm import Router
router = Router(
    model_list=[
        {
            "model_name": "chat",
            "litellm_params": {
                "model": "anthropic/claude-sonnet-4-5", # placeholder
                "api_key": os.environ["ANTHROPIC_API_KEY"],
            },
        },
        {
            "model_name": "chat-backup",
            "litellm_params": {
                "model": "openai/gpt-4o", # placeholder
                "api_key": os.environ["OPENAI_API_KEY"],
            },
        },
    ],
    fallbacks=[{"chat": ["chat-backup"]}],
    num_retries=1,
    timeout=20,
)
resp = router.completion(
    model="chat",
    messages=[{"role": "user", "content": "Summarize this ticket."}],
)
print(resp.choices[0].message.content)

What it does well: keys never leave your infrastructure, you can set per-model timeouts and cooldowns, and the same code path covers dozens of providers.

What tripped me up: the fallback fires per request, so during a long outage every request pays the primary's timeout first unless you configure cooldowns for failing deployments. Also, running the proxy server means you own its uptime, its database if you enable spend tracking, and its upgrades.

If you want a self-hosted gateway with fallbacks, budgets, and one OpenAI-compatible endpoint for every team, LiteLLM is the one that gives you that without sending traffic through a third party.

Takeaway: LiteLLM trades a service you rent for a service you run, and the timeout-before-fallback latency is the part to tune.

How does OpenRouter handle fallbacks?

OpenRouter is a hosted API that fronts many providers with one key and one billing relationship. You send an ordered list of models and it tries them in turn.

import os
from openai import OpenAI
client = OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key=os.environ["OPENROUTER_API_KEY"],
)
resp = client.chat.completions.create(
    model="anthropic/claude-sonnet-4.5", # placeholder
    messages=[{"role": "user", "content": "Summarize this ticket."}],
    extra_body={
        "models": [
            "anthropic/claude-sonnet-4.5",
            "openai/gpt-4o",
        ]
    },
)
print(resp.choices[0].message.content)

This is the least code, and one account replaces several provider invoices.

The drawbacks are real, though. Your prompts and your users' data pass through one more company, which matters if your contracts or compliance posture name specific subprocessors. The router itself becomes a single point of failure, so you have traded N providers' outages for N providers' outages plus one. And pricing includes a platform layer; check the current fee model on their site rather than trusting my memory.

If you want one key and one invoice and don't want to run anything, OpenRouter is the lowest-effort way to get multi-provider fallback.

Takeaway: OpenRouter removes operational work and adds a dependency you cannot debug from the inside.

When is a hand-rolled fallback enough?

If you call exactly two models and the request shapes are simple, a wrapper is shorter than the config for either tool. The key is classifying errors, not looping blindly.

import os
import anthropic
import openai
primary = anthropic.Anthropic(max_retries=1, timeout=20)
backup = openai.OpenAI(max_retries=1, timeout=20)
def ask(prompt: str) -> str:
    try:
        msg = primary.messages.create(
            model="claude-sonnet-4-5", # placeholder
            max_tokens=1024,
            messages=[{"role": "user", "content": prompt}],
        )
        return msg.content[0].text
    except (anthropic.APIConnectionError, anthropic.APITimeoutError, anthropic.InternalServerError):
        pass # 5xx, 529, network, timeout: fail over
    # 4xx (BadRequestError, AuthenticationError) propagate on purpose
    r = backup.chat.completions.create(
        model="gpt-4o", # placeholder
        messages=[{"role": "user", "content": prompt}],
    )
    return r.choices[0].message.content

The limitation: it does not scale. The third provider, tool calling with different schemas, and streaming mid-failure each add code, and that is the point where one of the two tools above starts paying for itself. Also note that a stream that dies halfway cannot be transparently resumed on another model; decide up front whether you discard the partial output and restart.

Takeaway: two models and no tool calls means write the wrapper; anything more and you are rebuilding a gateway.

Comparison

FAQ

Should I retry or fail over when my LLM API returns 529 overloaded?

Retry once with a short backoff, then fail over. Repeated retries during an overload extend the outage for your users and add load to the provider.

Does failing over to a different model change my output quality?

Yes. Prompts tuned for one model often behave differently on another, so run your evaluation set against the backup model before you rely on it in production.

Can I resume a streamed LLM response on another provider after it fails midway?

Not transparently. The backup model has no knowledge of the partial output, so you either restart the request or pass the partial text as context and accept a visible seam.

Bottom line

If you call two models and want to keep control, write the wrapper and classify errors carefully. If you are a platform team that needs shared keys, budgets, and many providers behind one endpoint, run LiteLLM. If you are a small team that wants one key and zero operations, accept the extra dependency and use OpenRouter.

Whichever you choose, test the fallback by blocking the primary's hostname in staging, because an untested failover path is the one that fails during the incident.

Related reading

• Your Postgres Migration Runner Needs a Retry Contract, Not Just a Lock Timeout
• Ship a Production RAG Chatbot in a Weekend with Claude, pgvector, and FastAPI
• Cursor vs GitHub Copilot vs Windsurf: Which AI Editor Fits How You Actually Work?