What should you base the choice of an LLM on?
Choose an LLM (large language model, a program that reads text and writes answers) by first deciding what you will use it for, then looking at its ratings and price for that use. Every model has tasks it handles well and tasks it handles poorly, so an overall ranking does not answer the question for your own work. Whether you want it to write code or to read a long contract changes both the ratings worth checking and the amount you will pay.
There are five steps.
- Decide the use
- Read the ratings for that use
- Filter by price and by how much the model can read at once
- Test it with your own text (in Japanese if you work in Japanese)
- Compare two models side by side
The first three steps narrow the field to two or three models, and the last two check how they do on your own work before you decide.
Step 1: Decide the use
Start by writing, in one sentence, the job you want the LLM to do. A form that shows what it reads and what it writes works best, such as "answer internal questions in a chat" or "read English papers and summarize the key points in Japanese." If you also note how much text it reads each time and how many times a day you will use it, you can plug those figures straight into the estimate in step 3.
Once the use is set, the ratings to check follow. On this site, each use page ranks models by a different rating. For example, the coding page ranks by human ratings on coding questions, while the page for reading documents ranks by accuracy on comprehension questions about a long story.
If you have two jobs, checking each job separately is more reliable than asking whether one model can handle both. A model that is good at conversation is not necessarily near the top for reading long documents.
Steps 2 and 3: Read the ratings, then filter by price and length
There are two kinds of rating. A human rating is a score built from people comparing two answers to the same question and picking one, so it reflects how pleasing a conversation or a piece of writing is. A benchmark (an exam that tests models on a shared set of problems) result is the share of problems with fixed answers that a model solved, so it reflects accuracy in math or programming. With either kind, a gap within the margin of error means the two are about the same. Epoch AI explains that benchmark scores can also shift a lot with small differences in the prompt or the evaluation settings.
Next, filter by conditions. API (the gateway for using an LLM from a program) prices are quoted per one million tokens, and the rate differs between what the model reads and what it writes. A token is a small unit that text is split into. The prices on the use pages are these API prices, separate from app subscriptions. How much a model can read at once (context length) matters when you work with long documents. Whether it can read images or use tools can also be a condition you cannot drop, depending on the use.
Setting the most you can pay before you start cuts down the time spent comparing rating numbers. If the rating gap is within the margin of error, pick the cheaper one; that is the basic rule.
ChatGPT and Claude are service names, and the models inside change
Names such as ChatGPT (OpenAI), Claude (Anthropic) and Gemini (Google) are in most cases the names of services. The models running inside them have their own names, and they are swapped when the developer releases new ones. For example, OpenAI replaced GPT-4 in ChatGPT with GPT-4o on April 30, 2025. At that point it kept GPT-4 in the API. Anthropic likewise uses Claude as the name for a family of models. For each model it lists the API name and a date until which, at the least, it will keep offering that model.
For this reason, compare ratings and prices by model name, not by service name. Even within the same service, a rating you read earlier is not necessarily the rating of the model inside it. The models you can pick in an app are not necessarily the same as those you can use through the API either, so check the name in the channel you actually use.
Think about which service to use for what in the same way, by use. There is no need to settle on a single service, and you can use different models for writing and for programming. For research, the first thing that matters is whether the service can search the web and show its sources. Then check the page for the smartest models, which ranks by human ratings on hard questions, and open and read the sources attached to the answers yourself.
Steps 4 and 5: Test with your own text, then compare two side by side
Once the ratings have narrowed things to two or three models, try them on text from your own work. Most benchmarks are built from English problems, so how well a model handles Japanese honorifics and phrasing can only be learned from ratings on Japanese questions or from testing it yourself.
Test as follows. Choose requests whose answers you can check for correctness yourself.
- Pick three to five requests from your everyday work (mix easy ones with hard ones)
- Give the same request text to both models
- Hide the names, line up the answers, and check accuracy, whether the instructions were followed, and readability
- Also write down how long each answer took and how many tokens it used
Hiding the names keeps you from being swayed by your impression of a familiar name. Arena's human ratings are collected the same way: people compare two answers with the names withheld, pick one, and only then see the names. Using the same format for your own requests lets you compare the rating numbers and your own impressions on the same scale.
Summary: when in doubt, go back to step 1
The order is the use, the ratings, the conditions, a test with your own text, and a side-by-side comparison of two. When you stay undecided for a long time, the use is usually vague, so go back to the one-sentence description of the job you want done.
Even as models change, these steps stay the same. When a new model comes out, comparing it in the same way on the three to five requests you kept lets you decide quickly whether to switch. The ratings on the use pages are re-imported once a week, so check the update date before you look.