A Human Rating Is a Score Built from Comparison Votes
A human rating is a score built from votes cast by people who compare two models' answers to the same question and pick the better one. The best-known example is Arena (formerly Chatbot Arena), which has users choose without knowing which model wrote which answer, and reveals the model names only after the vote. From the collected votes, it computes scores with the Bradley-Terry model, a statistical method that estimates strength from a record of wins and losses.
Its strength is that it captures qualities that problems with fixed answers struggle to measure, such as natural writing and clear explanations. Arena also produces scores by field, such as coding, creative writing, and questions in Japanese. The pages for choosing by use draw on these field scores to match each use.
Read a Gap as the Share of Times a Model Is Chosen
The gap between two models' scores means more than the scores themselves. A gap can be read as "the share of times the higher model is chosen when people compare the two." In the Elo rating conversion table used in chess, a gap of 50 gives about 57%, a gap of 100 gives 64%, and a gap of 200 gives 76%. The pages on this site convert gaps to shares on the same scale, and a gap of 0 means even odds. With a gap of 50, the lower model is still chosen 43 times out of 100.
The Limit of Being Pulled by Human Taste
Human ratings also absorb people's quirks of taste. In a 2024 analysis by the Arena operators, answer length moved votes the most, while formatting such as headers and bold had smaller effects. Scores with the effects of length and formatting removed were then produced, and this site uses those values. People comparing answers do not always notice factual errors, so for uses where accuracy matters, check benchmark values as well.