Lesson 5: Degradation

Circuit breakers

Retries handle a failure. They handle a sustained outage badly.

If a provider is down for ten minutes, every request retries five times with backoff before failing. Each document takes thirty seconds to fail instead of failing immediately, your workers spend the outage sleeping, and the provider receives a constant stream of requests it cannot serve. You have converted a fast, clear failure into a slow, expensive one.

A circuit breaker stops calling a service that is clearly failing, waits, and then tests carefully before resuming. The name comes from electrical circuits, and the analogy holds well: it trips to protect the system, and it must be reset deliberately rather than continuing to conduct.

It has three states.

Closed is normal operation. Calls pass through, and failures are counted. When failures exceed a threshold, the breaker opens.

Open means calls fail immediately without being attempted. No request is sent, no time is spent, and the failing service receives nothing. After a cooldown, the breaker moves to half open.

Half open allows a limited number of trial calls. If they succeed, the breaker closes and normal operation resumes. If any fails, it opens again and the cooldown restarts.

import time
from dataclasses import dataclass, field
from typing import Literal

BreakerState = Literal["closed", "open", "half_open"]


@dataclass
class CircuitBreaker:
    failure_threshold: int = 5
    cooldown_seconds: float = 30.0
    half_open_successes_required: int = 2

    _state: BreakerState = field(default="closed", init=False)
    _failures: int = field(default=0, init=False)
    _successes: int = field(default=0, init=False)
    _opened_at: float = field(default=0.0, init=False)

    def before_call(self) -> None:
        """Raise immediately if the circuit is open. Call this before attempting."""
        if self._state == "open":
            if time.monotonic() - self._opened_at >= self.cooldown_seconds:
                self._state = "half_open"
                self._successes = 0
            else:
                raise ProviderOverloadedError("circuit open, not attempting call")

    def record_success(self) -> None:
        if self._state == "half_open":
            self._successes += 1
            if self._successes >= self.half_open_successes_required:
                self._state = "closed"
                self._failures = 0
        else:
            self._failures = 0

    def record_failure(self) -> None:
        if self._state == "half_open":
            self._trip()
            return
        self._failures += 1
        if self._failures >= self.failure_threshold:
            self._trip()

    def _trip(self) -> None:
        self._state = "open"
        self._opened_at = time.monotonic()
        logger.error("circuit opened after %d failures", self._failures)

Getting the settings right matters more than the code. A threshold that is too low trips on ordinary noise and makes your system less available than the provider it is protecting. A cooldown that is too short reopens the circuit repeatedly. A cooldown that is too long leaves you down after the provider recovers. Start around five failures and thirty seconds, then adjust based on what you observe.

One breaker per dependency, not one globally. If your embedding provider is down, your database should still be reachable. Sharing a breaker across dependencies means one failure disables everything.

In production, use a library. pybreaker is the long-standing choice, and several newer libraries combine retry and circuit breaking in one decorator. As with retries, the value of building it once is understanding the state machine well enough to tune it.

Fallback chains

A circuit breaker tells you a service is unavailable. It does not tell you what to do instead. That is a design decision, and it should be made before the outage rather than during one.

A fallback chain is an ordered list of increasingly degraded options:

def summarize(document: Document) -> Summary:
    """Summarize with a fallback ladder, degrading rather than failing."""
    try:
        return summarize_with_primary(document)
    except (ProviderError, ProviderOverloadedError):
        logger.warning("primary model unavailable, trying secondary")

    try:
        return summarize_with_secondary(document)
    except (ProviderError, ProviderOverloadedError):
        logger.warning("secondary model unavailable, trying cache")

    cached = cache.get(document.content_hash)
    if cached is not None:
        return Summary.from_cache(cached, is_stale=True)

    return Summary.unavailable(
        document_id=document.id,
        reason="no summarization service available",
    )

Each step is worse than the one before and better than nothing. The final step is the important one: honest degradation, meaning a response that clearly says what could not be done rather than a fabricated one or an unhandled crash.

The is_stale flag on the cached result and the explicit unavailable result both exist so that callers, and eventually users, can tell the difference between a fresh answer and a degraded one. A fallback that silently returns worse data without saying so is worse than a failure, because nobody knows to distrust it.

Designing a degradation ladder in advance

Write the ladder down before you need it, because outages are a bad time to be deciding what your product should do.

For each capability, answer four questions. What is the ideal response? What is an acceptable degraded response? What is the minimum honest response? And what must never happen?

For the recipe extractor:

That last row is the one worth arguing about with your team, and it is the reason to write the table. In an outage, "return something plausible" is a tempting shortcut, and having previously agreed that it is forbidden is what stops it.

Two practical requirements. Every level must be observable, so your logs and metrics record which level served each request, or you will not know you have been running degraded for a week. And every level should be testable, meaning you can force it in a test rather than waiting for a real outage to find out whether the fallback works. A fallback path that has never executed is not a fallback, it is an untested branch.