Incident: A balancing loss that stopped balancing

Module 7 · Incident file · About 40 minutes

What happened

Hugging Face transformers ships the modelling code for Mixtral, the Qwen mixture-of-experts models and others. In its MoE code, the router computed raw scores and then overwrote the same variable, router_logits, with their softmax. On 30 March 2026, Ionut Anghelina reported the consequence in issue #45120 (github.com/huggingface/transformers/issues/45120): the value recorded for the balancing loss was already probabilities, and load_balancing_loss_func "applies softmax again". The result, in the issue's words, "flattens the routing probability distribution toward uniform", so the load-balancing loss "effectively provides no useful gradient signal."

The affected routers were Mixtral's, Qwen2-MoE's and Qwen3-VL-MoE's, plus models inheriting them: MiniMax, OLMoE, FlexOlmo, Qwen3-MoE, Qwen3-Next, Qwen3-Omni-MoE and Qwen3.5-MoE. It "only affects fine-tuning with output_router_logits=True" and "does not affect inference or pretrained model quality." No measured cost was published.

The same overwrite had been flagged in pull request #43542 on 28 January 2026, still open when checked in October 2026. Three fix pull requests appeared on 30 March. A maintainer explained on 9 April that, swamped by code-agent pull requests, "sometimes that means we do close legit fixes". Pull request #45131 was merged on 13 April 2026: two changed lines in each of 13 files.