A Test Built on Common Problems
A benchmark is a common test that has many LLMs solve problems with fixed answers so their results can be compared. There are types for each field: some have models solve math problems, some have them fix bugs in programs, and some have them answer expert-level questions. Because answers can be scored against the correct ones, models can be lined up on the same yardstick, unlike ratings in which people compare answers.
Reading a score as "how many out of 100 problems were solved" makes it concrete. A gap of 1 point in accuracy is one problem out of 100. On this site, task values are rescaled to 0 to 100 after removing the share that could be earned by guessing.
Two Reasons Scores Drift from True Ability
The first is contamination. When published problems slip into training text, the model memorizes the answers rather than how to solve them. A 2024 study that re-tested models on new problems built to mirror the grade-school math set GSM8K found some models whose accuracy fell by up to 8 points.
The second is the conditions of measurement. Even on the same benchmark, a different way of asking or different settings change the score. Epoch AI gives examples where its own scores differ from other reported scores and attributes this to differences in evaluation settings. Scores announced by a developer and scores measured by a third party are best not compared side by side as they are.
Reading the Page for Each Benchmark
The page for each benchmark on this site adds a ± range to each value and groups models whose gaps fall within the margin of error into the same rank range. Rather than deciding which is higher from the rank number alone, first check whether the gap goes beyond the range. On the pages for choosing by use, a benchmark is the ranking basis for some uses and appears as a reference column for others.