Topic 4: Protecting the service
16 min read·22 Sept 2026
Rate limiting with a token bucket
A rate limit caps how fast one client can use the server, so a runaway agent loop or a buggy script cannot starve everyone else. A token bucket is the usual algorithm: each client has a bucket holding up to capacity tokens; every request spends one; tokens refill at a steady rate. It allows short bursts (up to capacity) but not a sustained rate above the refill. An empty bucket means HTTP 429 Too Many Requests with a Retry-After header saying when to try again.
Two decisions matter more than the algorithm: what counts as "one client", and which requests cost a token.