Incident: A rope setting the loader never read

Module 4 · Incident file · About 40 minutes

What happened

On 23 July 2024 Meta released Llama 3.1, and Hugging Face's announcement said "All the variants support a context length of 128K tokens" and that the model "requires a minor modeling update to handle RoPE scaling effectively" (huggingface.co/blog/llama31, 23 July 2024). The update was a new block in each model's config.json: "rope_type": "llama3", with factor 8.0, low_freq_factor 1.0, high_freq_factor 4.0 and original_max_position_embeddings 8192, on top of rope_theta 500,000 (quoted from Meta's config in llama.cpp issue #8650).

The same day GitHub user Vaibhavs10 opened that issue, asking llama.cpp, a C/C++ inference engine, to add the scaling: "Without the modelling changes, the generation might look coherent, but they are far from great". Llama 3.1 files already converted for llama.cpp loaded and ran; the new block was simply not read. One user reported that for long context "the model isn't working unless I use 8000000 as RoPE base frequency for 48K context".

On 24 July GitHub user jmorganca opened pull request #8676, which computes the scaled frequencies at conversion and stores them in the model file: "this really improves results for context windows above 8192" (PR #8676). A tester summarising across the full 128K window wrote that "the output looks good whereas on master it outputs broken garbage". ggerganov, who maintains llama.cpp, noted the missing factors made generations "technically incorrect even for shorter contexts", and that "new GGUFs will have to be created either way". It merged on 27 July 2024. The cost: four days in which a newly released model ran degraded without an error, and every file converted before the fix had to be converted again.