Ship a Failover Lane for AI Calls in One Weekend (And Skip the Rest)

Saturday, 07:40. My side project's "summarize thread" button returned a 429. My retry loop made it worse. Every retry burned the same quota I was already out of.

By 08:30 I had a decision to make. Build a full AI gateway, or ship one honest thing by Sunday night. I shipped the one honest thing. A failover lane with a circuit breaker, a classifier for retryable errors, and five tests that run against fake upstreams.

No dashboard. No router config language. No Kubernetes. Here is the build log: the scope I kept, the code, the tests, and everything I deliberately skipped.

Step 1: Cut the scope before you write code

A weekend dies from ambition, not from bugs. So I wrote the cut list first.

Kept: one breaker per provider, retryable-error classification, a half-open probe, and a test harness. Skipped: distributed state, metrics export, prompt versioning, and cost accounting.

The skipped list is not a roadmap. It is a list of problems I do not have yet. One process, one API key, one user: that is my whole deployment.

Step 2: Decide what counts as a failure

This is the part most side projects get wrong. A 400 from a bad prompt is not an outage. Failing over on it just hides your bug behind another provider. So classify first. Only retryable failures should move traffic.

# lanes.py
RETRYABLE_STATUS = {408, 409, 425, 429, 500, 502, 503, 504}

def is_retryable(exc: Exception) -> bool:
    status = getattr(exc, "status_code", None)
    if status is None:
        # Network errors, timeouts, connection resets.
        return isinstance(exc, (TimeoutError, ConnectionError))
    return status in RETRYABLE_STATUS

401 and 403 are configuration bugs. 400 and 422 are prompt bugs. Send those to your logs, not to the fallback lane.

Step 3: Write the breaker (about 40 lines)

The breaker has three states. Closed means primary. Open means fallback only. Half-open means one probe gets through.

# breaker.py
import time
from lanes import is_retryable

class Breaker:
    def __init__(self, fail_threshold=3, cooldown_s=20.0):
        self.fail_threshold = fail_threshold
        self.cooldown_s = cooldown_s
        self.fails = 0
        self.opened_at = None

    def state(self):
        if self.opened_at is None:
            return "closed"
        if time.monotonic() - self.opened_at >= self.cooldown_s:
            return "half-open"
        return "open"

    def _trip(self):
        self.opened_at = time.monotonic()
        self.fails = 0

    def _reset(self):
        self.opened_at = None
        self.fails = 0

    def call(self, primary, fallback):
        state = self.state()
        if state == "open":
            return fallback(), "fallback_open"
        if state == "half-open":
            try:
                out = primary()
            except Exception as exc:
                if not is_retryable(exc):
                    raise
                self._trip()
                return fallback(), "fallback_probe_failed"
            self._reset()
            return out, "primary_recovered"
        try:
            out = primary()
        except Exception as exc:
            if not is_retryable(exc):
                raise
            self.fails += 1
            if self.fails >= self.fail_threshold:
                self._trip()
            return fallback(), "fallback_error"
        self._reset()
        return out, "primary"

The second return value is the only observability I allowed myself. It tells you which lane answered, and that is enough for a weekend.

Step 4: Test it with fake upstreams, not real ones

You cannot debug a breaker against a flaky provider. You can debug it against a fake that fails on command. Five scenarios cover almost everything I care about:

• Primary succeeds, breaker stays closed.
• Primary raises 429 three times, breaker opens, fallback answers.
• Primary raises 400, no failover, error surfaces to caller.
• Breaker is open, primary is never called at all.
• Cooldown elapses, probe succeeds, breaker closes again.

# test_breaker.py
import pytest
from breaker import Breaker

class Boom(Exception):
    def __init__(self, status_code):
        self.status_code = status_code

def primary_raises(status):
    def fn():
        raise Boom(status)
    return fn

def test_non_retryable_does_not_failover():
    b = Breaker(fail_threshold=1)
    with pytest.raises(Boom):
        b.call(primary_raises(400), lambda: "fallback")
    assert b.state() == "closed"

def test_opens_after_threshold():
    b = Breaker(fail_threshold=3)
    for _ in range(3):
        out, lane = b.call(primary_raises(429), lambda: "fallback")
        assert out == "fallback"
    assert b.state() == "open"

def test_open_skips_primary():
    b = Breaker(fail_threshold=1)
    b.call(primary_raises(503), lambda: "fallback")
    calls = []
    def spy():
        calls.append(1)
        return "primary"
    out, lane = b.call(spy, lambda: "fallback")
    assert lane == "fallback_open"
    assert calls == []

Scenario five needs a short cooldown. Set cooldown_s=0.05 and the half-open test runs in milliseconds.

Run it: pytest -q test_breaker.py. Five green tests, one afternoon, no API key.

Step 5: Pick a lane you can actually afford to leave on

A fallback lane only helps if it is cheap and available during someone else's outage. That rules out a second premium key for a hobby project.

This is where MonkeyCode fit my build. The operator states that the project offers free model access and a free server option, and that it is open source; I used it as the fallback lane and as the place to run the cron job. Treat those as operator-supplied claims, not as measured results.

As of this writing the operator also states a free token allowance in the millions, so check the current number on the project's own pricing page before you size anything around it. Allowances change.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The important design point is not the vendor. It is that your fallback must be a different failure domain from your primary. Two keys in the same account are one outage, not two.

What I skipped, and why it was fine

I skipped distributed state. The breaker lives in one process, so a deploy resets it. For a side project that runs one worker, a reset is a non-event.

I skipped per-endpoint breakers. My app has one model call path. Adding granularity before I had a second path would have been speculative abstraction.

I skipped metrics. The lane string already goes into my request log, and I read that log twice a week.

I also skipped prompt-level retries. If the model returns a wrong answer, a breaker will not save you. That is an eval problem, and it is next weekend's problem.

Should you build this?

Limitations, stated plainly

Free lanes have their own rate limits. A breaker does not buy you capacity. Half-open sends one probe. Under real concurrency you may want a token bucket there instead. And an in-memory breaker forgets everything on restart. If you need it to survive deploys, you need shared state, and that is a different weekend.

Who should not use this

Skip this approach if your fallback model has never passed your own eval set. Skip it if a regulator, not an engineer, decides where your data goes. Skip it if you are already at high request volume, because a single-process breaker will flap under load.

If you want to try the lane I used, start with a throwaway key and the five tests above. Get them green before you point real traffic at anything.

Sunday, 23:10. The button works, the tests pass, and the only new file is 40 lines long. The scope cut is what shipped the feature. Not the model.