Incident: Fine in bfloat16, NaN in float16

Module 3 · Incident file · About 40 minutes

What happened

On 12 January 2021 Suraj Patil (Hugging Face user valhalla) announced on the Hugging Face forum that "t5-small was the only T5 model that was working in fp16. The rest of the models produce nan loss/logits" (discuss.huggingface.co/t/3139). T5 is Google's encoder-decoder transformer, then "one most widely used model" in the transformers library, in his words. For "all the models and versions (v1, v1.1, mT5)", he wrote, "we get inf values in hidden_states after applying the final linear layer (wo)", the down projection of the feed-forward network, "which results in nan values in T5LayerNorm". For larger models the infinities also appeared "where we add the attn output with the hidden_states": the residual addition.

The fix, pull request #9487 ("[T5] enable T5 fp16", opened 8 January, merged 12 January 2021, github.com/huggingface/transformers/pull/9487), clamped the hidden states "to the max value of the current data type" after the feed-forward and both attention outputs. It did not hold. On 21 March 2021 Yuval Kirstain reported NaNs again with t5-large, from the norm's rsqrt line, adding "it happens in the last layers" (issue #10830); On 29 March stas00 wrote that "the problem is due to autocast in T5LayerFF", the feed-forward layer, and proposed turning mixed precision off there. In May 2023 Harry Mellor of Graphcore wrote that T5 "has only worked with hardware that supports bfloat16, the format that the model was originally trained with", measured the down projection's activations at a standard deviation of about 4,400, and noted that the residual additions overflowed too (graphcore.ai, 30 May 2023). The cost was not a crash. Users training or running T5 in float16 got NaN losses and logits, and could not have the "decent speed-up" the announcement promised, for more than two years on some hardware.