Can you decide on public benchmarks alone?
Use public benchmarks (tests that measure models on a shared set of problems) to narrow down the models you are torn between, and make the final decision on problems from your own work. Scores on public problems do not directly show how a model will do on your work.
There are three reasons. First, public problems may have been learned by the models. Second, public problems are tasks chosen by other people and differ from your own text and formats. Third, scores move with the way they are measured. As of October 2026, Epoch AI explains that small differences in prompts or evaluation settings can change results considerably.
Your own evaluation set (pairs of problems and correct answers built from your own work) avoids all three at once. Once built, you can reuse it every time you choose a model again.
Public problems may have been learned
The problems and answers of benchmarks are often published on the web. When the text used to train a model is collected from the web, those problems can get mixed in. A model that has memorized the problems scores high even without the ability to solve them. This is called benchmark contamination.
In a 2024 paper, some models dropped by up to 8 points in accuracy on new problems rebuilt in the same form and difficulty as a widely used set of grade-school math problems. If a model with a high public score does poorly on your own problems, suspect either contamination or a difference in the work.
Your own evaluation set is not on the web, so there is no worry that a model has learned it. Not publishing your evaluation set outside your organization is the condition for keeping its value. Even when sharing it inside your organization, limit who can see it, and do not put its questions as examples in outside documents or articles. A problem that has once gone outside may get mixed into the training of later models, so replace it with a new one.
Prepare 20-30 questions and the form of the correct answers
You can start with 20-30 problems. With only a few, chance differences sway you, and beyond 100 a person cannot finish scoring them. Choose them in proportion to your daily work, putting in more of the tasks you request most often. Anthropic also recommends making evaluations resemble the actual work and including edge cases.
- 70-80 percent ordinary requests
- 10-20 percent requests where the material is empty, irrelevant or too long
- A few ambiguous requests where even people would disagree
For each problem, write the question text, the material to pass, and the form of the correct answer in a table. For classification, write the correct label; for extraction, the items to pull out; for writing, the conditions for passing (such as "300 characters or fewer" or "no wrong dates"). A problem without a defined form of the correct answer leaves you unsure when scoring, and the scores waver.
For work that sorts inquiry emails into "order", "return", "fault", "billing" and "other", for example, pick 25 emails you actually received, remove names and numbers, and write down the correct category. Including 2-3 emails where even a person would hesitate over the category shows how a model answers when it is unsure.
Score by hand and check whether the difference is within the margin of error
A person does the scoring while looking at the form of the correct answer. Hide the model names and shuffle the answers before reading. For problems with a single correct answer, such as classification labels or numbers, simply count how many match.
Once you have scores, check whether the difference is within the margin of error. The margin of error is the size of difference that can arise by chance. With 20-30 problems, a difference of a few problems can happen by chance. If you give each question twice, you find problems where even the same model gives different answers. Read a difference smaller than that wobble as almost the same.
For writing problems, scores tend to change depending on who scores them. Write the scoring criteria in 3-4 lines and apply the same criteria to every problem, and if possible have two people score and discuss the problems where they split. Even when scoring alone, rescoring half of them after a few days shows how much your own criteria wobble.
Once you know the difference is small, decide on conditions other than the ratings. Choose the cheaper one, the faster one, or the one easier to build into the systems you have. Paying the price difference for the higher score makes sense only when the difference is clear.
Redo it whenever the model changes
An evaluation set is not finished once it is built. Redo it at these times.
- When a developer releases a new version
- When the model inside the app you use changes
- When the content of your work or the form of your material changes
Each time you redo it, add 1-2 requests that did not go well. Do not throw away all the old problems; keep about half. The kept problems let you compare last time's scores with this time's, and the new ones let you follow changes in your work. Record the results in the table with the date and model name.
Put the problems and answers in a single table, in a form anyone can score with the same procedure. Even if the person in charge changes, you can keep comparing with the same yardstick and line up results from half a year ago.
Summary - narrow with public scores, decide with your own problems
Public benchmarks may have been learned and differ from your work. Narrow the field to two or three with the ratings on the use pages, and decide with your own evaluation set. Use 20-30 problems, write the form of the correct answer first, and have a person score with the names hidden. If the difference is within the margin of error, choose by price or speed.
For work with fixed correct answers, such as classification and extraction, it is quick to take the models narrowed down on the classification and extraction page and compare them by the number of matches on your own problems. Keep the evaluation set private, and line the models up again on the same problems whenever they change.