What does a benchmark score represent?
An LLM benchmark score is the share of a shared set of problems that the model solved. On raw accuracy, a score of 70 means it solved the equivalent of 70 problems out of 100, and a 3-point gap between two models is a difference of 3 problems in 100.
That gap, however, is not necessarily a gap in ability. The number of problems is limited, and even the same model answers a little differently each time, so every score has a margin of error. If the gap falls within that margin, read the two as about the same.
With multiple-choice problems, also keep in mind the share a model gets right by answering at random. With four options, even pure guessing hits one problem in four. For benchmarks whose guessing score is known, the task values on this site remove that share and rescale to 0 to 100, so 0 equals guessing and 100 is a perfect score. The sections below explain three things in turn: the margin of error, problems leaking into training, and test conditions.
Why is there a margin of error?
There are two reasons for the margin of error. The first is the number of problems. A benchmark's problems are only a portion drawn from the whole pool of problems that could test the ability being measured. There is a 2024 paper on how to add error bars to evaluations (Evan Miller). It too treats evaluation problems as a drawn portion and presents formulas for analyzing the results.
The fewer the problems, the wider the margin. For example, if a model with accuracy around 70% is tested on 100 problems, the range that chance alone can move it (the 95% range) is roughly ±9 points. With 1,000 problems it shrinks to about ±3 points. You have to quadruple the problems just to halve the margin.
The second reason is variation in the answers. An LLM writes a slightly different answer to the same problem each time, so a score from a single run depends on luck. Epoch AI tests many models 16 times on expert-level science multiple-choice questions and uses the average as the score (as described in October 2026).
Reading the gap: overlapping ± ranges mean about the same
The pages on this site add a ± range to each score. A gap where the two ranges overlap is within the margin of error (about the same), and a gap where they do not overlap is hard to explain by chance alone. The △ mark appears in the "Best-rated model at each price" table on the use pages. It marks the pricier row whose range overlaps with that of the nearest row below it without △. In the bar charts on the pages, the end of a bar is the value and the thin line is the ± range.
Whether the ranges overlap, however, is not a sharp line between black and white. The smaller the overlap, the more likely the gap is real; the larger the overlap, the more likely the order flips when you measure again. If you are torn between a row marked △ and the row below it without △, pick the cheaper row without △ or the one that did better when you tried it on your own work.
Read the rank numbers the same way. A page shows a rank as a range such as "2 to 5" because models whose ranges overlap are given the same range of ranks. Even second and fourth place differ very little if their ranges overlap.
Contamination: when models learn the benchmark problems
When benchmark problems, or text closely resembling them, get mixed into training data, a model memorizes answers rather than learning how to solve them. This is called contamination. Published problems get quoted on the web, so they have chances to end up in training data.
GSM8K is a benchmark of grade-school math word problems. A 2024 paper built 1,000 new problems (GSM1k) matched to it in format and difficulty, and retested many models. Some models lost up to 8 points of accuracy, and several families of models showed signs of having memorized the problems. Most frontier models, on the other hand, showed almost no such signs.
The degree of contamination differs from model to model and cannot be told from the score alone. Do not decide on scores from problems published long ago; also look at scores on newly built problems and on benchmarks that keep their problems private. In this site's benchmark list, a benchmark that has a successor is shown with the successor's name.
The same benchmark can score differently under different conditions
Even for a benchmark with the same name, the score changes with who ran it and under what conditions. Epoch AI explains that small differences in the prompt or the evaluation settings can change results a lot. Its example is one model's score on expert-level science multiple-choice questions. The developer reported 65%, but the average of Epoch AI's 16 runs was about 55% (as described in October 2026).
Differences in conditions include how the prompt is written, how the answer format is specified, and the reasoning setting (how long the model spends thinking through steps before it answers). A model that fails to write its answer in the required format and loses marks in grading can even score below random guessing. The comparison table on this site also lists values for each reasoning setting.
For this reason, compare scores only with values of the same kind. This site's benchmark list shows two kinds separately. "Task results" are results that an organization compiling evaluations ran itself under the same conditions. "Reported by others" collects values published by the benchmark's creators and other organizations.
Summary: four questions for reading a score
Use a benchmark score only after answering these four questions.
- What do the problems measure (are they close to your use)?
- Does the gap exceed the margin of error?
- Could the problems have leaked into training?
- Were the scores measured under the same conditions?
Only a gap that passes all four is a reason to choose. To check math ability, look at the math and logic page; for depth of knowledge, the page for expert questions, where the rating used for ranking sits next to the reference ratings.