Circuit Breakers for AI: How to Fail Without Failing the User

The spinner of death

Here's the failure a circuit breaker exists to prevent. Your model provider starts timing out. Without a breaker, every incoming request still gets dispatched to the dying dependency. Each one occupies a worker, waits out the full timeout, fails, and frees the worker, just in time for the next doomed request. Your worker pool fills with requests that are guaranteed to fail, latency climbs to the timeout ceiling for everyone, and users stare at a spinner for thirty seconds before getting an error. You've converted a degraded dependency into a total outage, and spent maximum latency and maximum resources to produce zero successful responses.

A circuit breaker breaks that loop. After it observes enough failures, it stops calling the dependency at all, it "opens", and immediately returns a fallback. Requests fail (or degrade) in milliseconds instead of seconds. The dying dependency gets breathing room to recover instead of a continuous barrage. And critically, your worker pool stays free to serve the parts of your product that don't depend on the model.

The three states, and why half-open is the one that matters

The circuit breaker pattern (popularized by Martin Fowler and Michael Nygard's Release It!) has three states:

• Closed, normal operation. Requests flow to the dependency; the breaker counts failures.
• Open, tripped. Requests fail fast or fall back immediately, without touching the dependency. The breaker stays open for a cool-down period.
• Half-open, after the cool-down, the breaker lets a small number of trial requests through. If they succeed, it closes (recovery confirmed). If they fail, it re-opens and waits again.

For a production-grade reference implementation of these transitions and fallbacks, Microsoft's Cloud Design Patterns: Circuit Breaker is a solid guide.

Half-open is the state most home-grown breakers get wrong. If you skip it and slam straight from open to closed after a timer, the instant the cool-down expires you dump your full traffic onto a dependency that may have only partially recovered, and immediately knock it back down. That's a thundering herd dressed up as recovery. Half-open is the controlled probe that prevents it: a trickle of requests confirms health before you trust the dependency with real load.

A breaker built for AI endpoints

Below is a compact breaker. What makes it AI-specific is the fallback ladder it feeds, not the state machine itself.

import time

class CircuitBreaker:
    def __init__(self, fail_threshold=10, cool_down=15, half_open_trials=3):
        self.fail_threshold = fail_threshold
        self.cool_down = cool_down
        self.half_open_trials = half_open_trials
        self.state = "closed"
        self.failures = 0
        self.opened_at = 0
        self.trial_successes = 0

    def _to_half_open(self):
        self.state = "half_open"; self.trial_successes = 0

    def allow(self):
        if self.state == "open":
            if time.time() - self.opened_at >= self.cool_down:
                self._to_half_open(); return True   # probe
            return False                            # fail fast -> fallback
        return True

    def record(self, ok):
        if self.state == "half_open":
            if ok:
                self.trial_successes += 1
                if self.trial_successes >= self.half_open_trials:
                    self.state, self.failures = "closed", 0
            else:
                self.state, self.opened_at = "open", time.time()
        elif ok:
            self.failures = 0
        else:
            self.failures += 1
            if self.failures >= self.fail_threshold:
                self.state, self.opened_at = "open", time.time()

The fallback ladder: degrade, don't die

An open breaker is only as good as what it falls back to. For AI features you almost always have cheaper options than "show an error." Order them by cost and quality:

def answer(query, breaker):
    if breaker.allow():
        try:
            resp = primary_model(query, timeout=8)   # premium model
            breaker.record(ok=True)
            return resp
        except (Timeout, ProviderError):
            breaker.record(ok=False)
            # fall through to the ladder

    # Breaker open OR primary just failed, degrade gracefully:
    if cached := semantic_cache.lookup(query):
        return cached.with_note("cached")            # 1. cached answer
    if small := cheap_fast_model(query, timeout=4):
        return small.with_note("lite mode")          # 2. smaller model
    return canned_help_response(query)               # 3. honest fallback

A cached or smaller-model answer in lite mode is a vastly better experience than a thirty-second hang. The fallback ladder is what turns "the AI is down" into "the AI is a little less smart right now", and that difference is the whole point of the pattern.

Why an untested breaker is worse than none

A circuit breaker has exactly two ways to be wrong, and both are damaging:

Too sensitive, a low failure threshold or short window trips the breaker on normal latency variance. Now you're serving degraded fallback responses during a perfectly healthy provider blip, throwing away quality for no reason and confusing users with "lite mode" when nothing was actually broken.

Too slow, a high threshold or long window means the breaker doesn't trip until the cascade is well underway. By the time it opens, your pool is already exhausted and the outage already happened. The breaker fired the alarm after the building burned down.

The only way to tune the threshold, cool-down, and half-open trial count correctly is to watch the breaker behave under a real fault at real concurrency. Tuning these by intuition is guessing, and the cost of guessing wrong is either chronic false degradation or an outage the breaker was supposed to stop.

There's a third subtlety that bites in AI stacks specifically: the failure threshold should usually key on latency, not just hard errors. A model provider that's degraded rarely returns clean 500s, it returns slow, timing-out responses that eventually succeed. If your breaker only counts exceptions, it never trips while every request crawls toward the timeout ceiling and your pool quietly fills. Count slow responses (anything past, say, your p99.9 latency) as failures so the breaker opens on the degradation pattern that actually precedes an AI outage, not just on the clean errors that rarely arrive in time to help. Per-dependency breakers matter too: if you call several providers or several models, one breaker for all of them will trip the healthy paths when only one is sick, so scope a breaker to each downstream you depend on.

Where the breaker lives matters

A circuit breaker can sit in three places, and the choice changes both what it protects and how you test it. The most common is a per-instance, in-process breaker: each application server keeps its own failure counts. It's simple and fast, but it has blind spots, each instance has to learn the dependency is down independently, so during a fault you get N instances all discovering the problem separately, and a low-traffic instance may never see enough failures to trip at all.

A shared/distributed breaker keeps state in something like Redis so all instances trip together the moment the dependency degrades. It reacts faster and more uniformly, but it adds a dependency of its own, and if that store is slow, you've put a synchronous call in your hot path. The third option, common in service meshes and sidecars (Envoy, Istio), is an infrastructure-level breaker that trips outside your application code entirely, which is clean but harder to wire into application-specific fallbacks.

For AI endpoints, the per-instance breaker with a fast fallback ladder is usually the right default, provider faults are global, so every instance will see them quickly, and you avoid coupling your resilience mechanism to a shared store. But whichever you choose, the load test has to match: a distributed breaker needs the test to confirm all instances trip in unison, while a per-instance breaker needs the test to confirm that even low-traffic instances trip before their pool exhausts. Test the breaker where it actually runs.

Load-test the breaker, not just the happy path

The test that matters injects a provider fault mid-load and asserts three things: the breaker trips fast enough that the pool never saturates, the fallback ladder serves users during the open period, and the half-open probe recovers cleanly without a re-collapse when the dependency returns.

# Fault-injection test for breaker behavior under load
async def breaker_test(concurrency=2000):
    metrics = await drive_load(
        concurrency=concurrency,
        timeline=[
            ("healthy",   0,  30),       # establish baseline
            ("provider_down", 30,90),   # 60s hard fault
            ("recovered", 90,150),      # dependency returns
        ])

    fault = metrics.window("provider_down")
    recov = metrics.window("recovered")

    # 1. Pool must NOT saturate during the fault (breaker did its job)
    assert fault.max_pool_utilization < 0.9, "breaker too slow; pool saturated"
    # 2. Users still got SOMETHING, fallback ladder served them
    assert fault.served_rate > 0.95, "fallback not serving during open state"
    # 3. p99 during fault must be fast-fail, not timeout-bound
    assert fault.p99_ms < 500, "requests waiting on dead dependency"
    # 4. Clean recovery: no re-collapse when traffic resumes (half-open works)
    assert recov.error_spike < 0.05, "thundering herd on recovery; half-open broken"
    print("breaker trips fast, degrades gracefully, recovers cleanly")

The bottom line

Without a circuit breaker, a degraded model API costs you maximum latency and maximum resources to deliver zero successful answers, the spinner of death. With one, the same fault costs you a fast, cheap, degraded response and a pool that stays free for the rest of your product. Provider outages aren't hypothetical, OpenAI's June 2025 incident ran 34 hours. The pattern is settled; the wins are in the details: a half-open probe that recovers without a herd, and a fallback ladder, cache, then small model, then honest message, that keeps users served. And none of it counts until you've watched it behave under an injected fault at production concurrency. A breaker you haven't load-tested is a guess about your worst day.