CourseLarge Language Models: The Complete Course · Module 4: Prompt Engineering · part 33 of 103
Part 33 · Module 4: Prompt Engineering

Incident: Same models, three harnesses, three leaderboards

13 min read·13 Oct 2026
Module 4 · Incident file · About 30 minutes

What happened

OPEN LLM LEADERBOARD, 2023 leaderboard LLaMA 65B 0.488 paper LLaMA 65B 0.637 23 June post 3 implementations rescoring updated Harness Twitter: why is the leaderboard lower?
The leaderboard's number (red) disagreed with the paper's (blue); the 23 June 2023 post traced it to the implementation (amber), and every model was rescored (green).

In June 2023 Hugging Face's Open LLM Leaderboard showed Meta's LLaMA 65B with an MMLU score of 0.488. The LLaMA paper had reported 0.637 on the same benchmark. In the words of the leaderboard team, "an interesting discussion arose on Twitter following the release of Falcon": the community "was surprised" that the leaderboard's numbers "were significantly lower than the numbers in the published LLaMa paper." On 23 June 2023 Clémentine Fourrier, Nathan Habib, Julien Launay and Thomas Wolf published what they found ("What's going on with the Open LLM Leaderboard?", Hugging Face blog).

MMLU, SAME MODELS, THREE IMPLEMENTATIONS model Original HELM Harness llama-65b 0.636 0.637 0.488 falcon-40b 0.558 0.571 0.527 llama-30b 0.584 0.583 0.457
The same three models score far lower under the Harness implementation (red) than under the original code or HELM, from the Hugging Face post.

MMLU is a multiple-choice test. The team ran three implementations of it: the original code from its authors at Berkeley, Stanford's HELM, and the EleutherAI Harness that the leaderboard used. All three use 5 examples. LLaMA 65B scored 0.636, 0.637 and 0.488. Comparing the leaderboard's 0.488 with the paper's 0.637, they wrote, compares "numbers which are not at all comparable".

What it cost was the leaderboard's meaning. The differences "even change the ranking order of the models on the leaderboard!" The team wrote that they were "currently updating the full leaderboard with the updated version of the EleutherAI Eval Harness": every model was rescored.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside