Incident: A serving bug dropped the best token
Module 6 · Incident file · About 35 minutes
What happened
On 17 September 2025 Anthropic published "A postmortem of three recent issues" on its engineering blog. It opens: "Between August and early September, three infrastructure bugs intermittently degraded Claude's response quality." No request failed. Users saw worse answers, and the company said plainly that none of it was deliberate: "We never reduce model quality due to demand, time of day, or server load. The problems our users reported were due to infrastructure bugs alone."
The first bug was a routing error: from 5 August "some Sonnet 4 requests were misrouted to servers configured for the upcoming 1M token context window". A load-balancing change on 29 August made it worse; "at the worst impacted hour on August 31, 16% of Sonnet 4 requests were affected", and routing was "sticky", so a user who hit the wrong server tended to stay on it. The fix went out on 4 September. The second, from 25 August, was a TPU misconfiguration that "occasionally assigned a high probability to tokens that should rarely be produced given the context, for example producing Thai or Chinese characters in response to English prompts". It was rolled back on 2 September.
The third is this file's subject. "On August 25, we deployed code to improve how Claude selects tokens during text generation. This change inadvertently triggered a latent bug in the XLA:TPU compiler." The post explains the background: "In December 2024, we discovered our TPU implementation would occasionally drop the most probable token when temperature was zero. We deployed a workaround to fix this case." The August change removed that workaround and exposed a deeper problem in the approximate top-k operation, "a performance optimization that quickly finds the highest probability tokens". The approximation "sometimes returned completely wrong results, but only for certain batch sizes and model configurations". It was rolled back for Haiku 3.5 on 4 September and for Opus 3 on 12 September.