What should you test before paying?
What you test before paying is questions from your own work. Prepare 5-10 questions, give the same questions to two models, and score the answers on axes you have set. Finally, compare the results with the ratings on the use page. Three to five questions are enough just to get a first feel, but if you are deciding what to pay for, use 5-10 so that chance differences do not sway you.
Public ratings summarize results on other people's questions or on shared problems. They cannot fully tell you how a model does with the text and phrasing of your own work. Even with a few questions, results you have tested yourself let you read the rating figures in terms of your own work.
Narrow the test to two models. Comparing three or more at once adds to the scoring work and makes the differences harder to remember. Look at ratings and prices on the use page, pick the two you are torn between, and then start.
Pick 5-10 questions from your own work
Pick questions from requests you actually want to make in your daily work. Mix easy and hard ones, and choose questions whose answers you can check yourself. If you cannot tell whether an answer is correct, you cannot decide which one is better.
- Include both short questions and requests to read a long document
- Include at least one piece of Japanese text where honorifics or phrasing matter
- Include one request that has not gone well in the past
In sales work, for example, the questions might be "pick out three next actions from these meeting notes", "write a polite email telling a client that delivery will be late" and "check whether the total of this quote is right". In each case you can judge for yourself whether the answer is good.
Once you have written the text of a question (the prompt), do not change it. If you change the wording, you cannot tell whether a difference in the answers comes from the models or from the wording. Write down the conditions for a good answer in a line or two before testing as well. If you set the conditions after seeing the answers, you will be pulled toward the answer you read first.
Give the same questions to two models
Give the two models exactly the same questions, word for word. Start a new conversation for each question so that earlier exchanges do not mix into the answer. In some apps, a feature that remembers past conversations affects the answers, so it is best to turn it off while testing.
If you test through the API (the gateway for using an LLM from a program), you can write a system prompt (instructions placed before the conversation). Use the same one for both models, or leave both empty. Writing instructions for only one of them makes the conditions unequal.
Hide the names when scoring. Paste the answers into a separate document and shuffle them so that you cannot tell which model wrote which before reading. Arena also collects its human ratings with the names hidden, and as of October 2026 it also offers a way to pick two models yourself and put them side by side. To shuffle quickly, swap columns in a spreadsheet or ask someone to put labels on the answers.
Score on correctness, following instructions, Japanese and speed
There are four axes for scoring. Give each a small score, such as 0-2 points, and add a word on the reason.
- Correctness: are there no errors in facts or calculations?
- Following instructions: did it keep to the length, format and restrictions?
- Natural Japanese: do the honorifics and phrasing read smoothly?
- Speed: is the time until the answer finishes short enough to wait for?
For an answer that writes an apology email, for example, judge wrong dates or amounts under correctness, and whether it kept to the specified length under following instructions. Judge excessive humility or unnatural honorifics under natural Japanese. If you avoid counting the same flaw on two axes, the totals do not tilt.
As a rule, do not choose a model that loses on correctness, even if it wins on the other axes. With an answer that contains errors, the work of fixing it outweighs the gain in readability. Speed matters if you use the model for conversation, but matters little for work you have it process in bulk. Decide beforehand which axes to weigh most for your use.
Check the terms of free trials on each developer's pricing page
Developers change how much you can try for free and the limits on use. Do not memorize the figures; check each developer's pricing page before testing. An app's monthly fee and the API price per million tokens are different things, so the page to read depends on which one you plan to use. A token is a unit that text is cut into.
On a free tier, what you enter may be handled differently from a paid tier. As of October 2026, Google states on its pricing page that content entered on the free tier of the Gemini API is used to improve its products, and content on the paid tier is not. Do not test with text that cannot leave your organization on a free tier. Rewrite the test questions into a form that would cause no trouble if someone else saw them.
How your data is handled depends on the service and the type of contract. When in doubt, read the terms of use and the data handling page of the gateway you use before entering anything.
Summary - compare the results with the page ratings
If your scores point the same way as the ratings on the page, you can take those ratings as applying to your work too. If they point the opposite way, put your own results first for your own work. With only 5-10 questions, however, a difference of one or two questions can happen by chance. When the difference is small, choose the cheaper one or the one you are used to.
Keep the questions you used and the score table. A line per question with the scores for the two answers and anything you noticed is enough. When a new version comes out or you consider switching, putting the models side by side again on the same questions lets you compare in a short time. If Japanese quality matters most to you, put the ratings on the page for using models in Japanese next to your own scores to check.