The Margin of Error Marks "About the Same"
Being within the margin of error means the gap between measured values is small enough to be explained by chance, so you cannot say which is higher. On the pages for choosing by use, rows whose rating gap with a cheaper model falls within the margin of error carry a △. A row with △ costs more but rates about the same. When ratings are about the same, choose the cheapest row in the table.
The page for each benchmark follows the same idea: it adds a ± range to each value and groups models whose gaps fall within the margin of error into the same rank range. That is why a rank appears as a range rather than a single number.
Why Gaps Wobble
The problems in a benchmark are only a sample drawn from all the problems that could be asked. The fewer the problems, the more a score moves depending on whether a model happened to get many it is good at. It is the same kind of wobble as one person scoring differently on two sittings of the same kind of test. A 2024 paper on the statistics of evaluations treats them as experiments and advises reporting a range of uncertainty for the gap between two models as well.
Human ratings work the same way: the fewer the votes a model has, the wider the ± range on its score. There are also ways to reduce the wobble; Epoch AI has models solve the same problems many times and takes the average.
Read the Size of the Gap, Not Just the △
The inside and outside of the margin of error are not separated by a sharp boundary. A gap that only just exceeds the range means "probably higher" at most, and the order may flip on re-measurement. Conversely, a gap inside the range is not necessarily zero. A gap several times the range can be taken as a real difference. Rather than deciding by the △ alone, looking at the size of the gap together with the price difference leads to fewer wrong choices.