Task results
Following format instructions given in Japanese (MIFEvalJa, measured with the English tasks)
The task of giving format instructions in Japanese, such as "use bullet points" or "include a given word", and checking automatically whether they were followed, measured as part of the English task set.
How to read this table
- Higher rows are rated higher. A range like “1–2” means the gap is within the margin of error.
- The score is the share of tries on a shared set of tasks that the model completed (%).
- A gap of 3 means 3 successes out of 100 tries.
- Prices are a guide per million tokens, reading then writing.
- Length is how much text the model can read at once (tokens).
| Rank | Model | Developer | Success rate (%) | Evaluated | Japanese (human rating) | Input $ /1M | Output $ /1M | Context | Supports | Weights | Released | Price and specs |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Gemma 4 31B | Rating 76.5 | — | — | — | 262.1K | Google Released 2026-04 Price — Length 262.1K | |||||
| 2 | Gemma 4 26B A4B | Rating 72.7 | — | $0.15 | $0.6 | 262.1K | Google Released 2026-04 Price $0.15 / $0.6 Length 262.1K | |||||
| 3 | GPT-5 | OpenAI | Rating 72.1 | 1431rank 17–93 | $1.25 | $10 | 400K | OpenAI Released 2025-08 Price $1.25 / $10 Length 400K | ||||
| 4 | o3 | OpenAI | Rating 71.2 | 1424rank 22–95 | $2 | $8 | 200K | OpenAI Released 2025-04 Price $2 / $8 Length 200K | ||||
| 5 | GPT-5 mini | OpenAI | Rating 70.3 | 1364rank 63–146 | $0.25 | $2 | 400K | OpenAI Released 2025-08 Price $0.25 / $2 Length 400K | ||||
| 6 | o3-mini | OpenAI | Rating 67.7 | 1315rank 101–175 | $1.1 | $4.4 | 200K | OpenAI Released 2025-01 Price $1.1 / $4.4 Length 200K | ||||
| 7 | NVIDIA Nemotron 3 Super 120B A12B | NVIDIA | Rating 66.0 | — | — | — | 262.1K | NVIDIA Released 2026-03 Price — Length 262.1K | ||||
| 8 | GPT-5.4 | OpenAI | Rating 65.1 | 1480rank 2–50 | $2.5 | $15 | 1.05M | OpenAI Released 2026-03 Price $2.5 / $15 Length 1.05M | ||||
| 9–10 | Qwen3.5 122B A10B | Alibaba | Rating 64.5 | 1361rank 60–154 | $0.4 | $3.2 | 262.1K | Alibaba Released 2026-02 Price $0.4 / $3.2 Length 262.1K | ||||
| 9–10 | Qwen3.5 27B | Alibaba | Rating 64.5 | 1339rank 71–166 | $0.3 | $2.4 | 262.1K | Alibaba Released 2026-02 Price $0.3 / $2.4 Length 262.1K | ||||
| 11 | GPT-OSS 120B | OpenAI | Rating 61.0 | 1325rank 95–166 | — | — | 128K | OpenAI Released 2025-08 Price — Length 128K | ||||
| 12 | GPT-OSS-Swallow-120B-RL-v0.1 | Swallow (Science Tokyo) | Rating 60.5 | — | — | — | — | — | — | Swallow (Science Tokyo) Released — Price — Length — | ||
| 13 | Qwen3.5 35B A3B | Alibaba | Rating 59.9 | 1315rank 93–183 | $0.25 | $2 | 262.1K | Alibaba Released 2026-02 Price $0.25 / $2 Length 262.1K | ||||
| 14 | Qwen3 Next 80B A3B | Alibaba | Rating 59.0 | 1293rank 112–202 | $0.5 | $6 | 131.1K | Alibaba Released 2025-09 Price $0.5 / $6 Length 131.1K | ||||
| 15 | GPT-OSS-Swallow-120B-SFT-v0.1 | Swallow (Science Tokyo) | Rating 56.4 | — | — | — | — | — | — | Swallow (Science Tokyo) Released — Price — Length — | ||
| 16 | GPT-OSS 20B | OpenAI | Rating 55.8 | 1256rank 140–216 | — | — | 131K | OpenAI Released 2025-08 Price — Length 131K | ||||
| 17 | Qwen3-Swallow-32B-RL-v0.2 | Swallow (Science Tokyo) | Rating 52.9 | — | — | — | — | — | — | Swallow (Science Tokyo) Released — Price — Length — | ||
| 18 | GPT-OSS-Swallow-20B-RL-v0.1 | Swallow (Science Tokyo) | Rating 51.7 | — | — | — | — | — | — | Swallow (Science Tokyo) Released — Price — Length — | ||
| 19 | LLM-jp-4 32B A3B | LLM-jp | Rating 50.9 | — | — | — | — | — | — | LLM-jp Released — Price — Length — | ||
| 20 | Qwen3 235B A22B (thinking) | Alibaba | Rating 49.4 | 1376rank 49–142 | — | — | 262.1K | Alibaba Released 2025-07 Price — Length 262.1K | ||||
| 21 | GPT-OSS-Swallow-20B-SFT-v0.1 | Swallow (Science Tokyo) | Rating 48.8 | — | — | — | — | — | — | Swallow (Science Tokyo) Released — Price — Length — | ||
| 22 | Qwen3.5 9B | Alibaba | Rating 48.3 | — | — | — | 262K | Alibaba Released 2026-02 Price — Length 262K | ||||
| 23 | Qwen3-Swallow-32B-SFT-v0.2 | Swallow (Science Tokyo) | Rating 47.7 | — | — | — | — | — | — | Swallow (Science Tokyo) Released — Price — Length — | ||
| 24 | Llama 3.3 70B (instruct) | Meta | Rating 44.5 | 1214rank 176–224 | — | — | 131.1K | Meta Released 2024-12 Price — Length 131.1K | ||||
| 25 | LLM-jp-4 8B | LLM-jp | Rating 43.9 | — | — | — | — | — | — | LLM-jp Released — Price — Length — | ||
| 26 | Qwen3-Swallow-32B-CPT-v0.2 | Swallow (Science Tokyo) | Rating 43.6 | — | — | — | — | — | — | Swallow (Science Tokyo) Released — Price — Length — | ||
| 27 | Qwen3-Swallow-30B-A3B-RL-v0.2 | Swallow (Science Tokyo) | Rating 43.3 | — | — | — | — | — | — | Swallow (Science Tokyo) Released — Price — Length — | ||
| 28–29 | NVIDIA-Nemotron-Nano-9B-v2-Japanese | NVIDIA | Rating 43.0 | — | — | — | — | — | — | NVIDIA Released — Price — Length — | ||
| 28–29 | Qwen3 235B A22B (instruct) | Alibaba | Rating 43.0 | 1392rank 44–119 | — | — | 128K | Alibaba Released 2025-07 Price — Length 128K | ||||
| 30 | GPT-4.1 | OpenAI | Rating 41.6 | 1347rank 83–150 | $2 | $8 | 1.05M | OpenAI Released 2025-04 Price $2 / $8 Length 1.05M | ||||
| 31 | Gemma 4 E4B | Rating 41.0 | — | — | — | 131.1K | — | — | Google Released — Price — Length 131.1K | |||
| 32–34 | Qwen3.5 4B | Alibaba | Rating 40.4 | — | — | — | 262.1K | Alibaba Released 2026-02 Price — Length 262.1K | ||||
| 32–34 | Qwen3-Swallow-30B-A3B-SFT-v0.2 | Swallow (Science Tokyo) | Rating 40.4 | — | — | — | — | — | — | Swallow (Science Tokyo) Released — Price — Length — | ||
| 32–34 | Qwen3-Swallow-8B-RL-v0.2 | Swallow (Science Tokyo) | Rating 40.4 | — | — | — | — | — | — | Swallow (Science Tokyo) Released — Price — Length — | ||
| 35 | Qwen3-Swallow-8B-SFT-v0.2 | Swallow (Science Tokyo) | Rating 39.5 | — | — | — | — | — | — | Swallow (Science Tokyo) Released — Price — Length — | ||
| 36 | Qwen3-Swallow-30B-A3B-CPT-v0.2 | Swallow (Science Tokyo) | Rating 39.0 | — | — | — | — | — | — | Swallow (Science Tokyo) Released — Price — Length — | ||
| 37 | Qwen3 Next 80B A3B (instruct) | Alibaba | Rating 38.4 | 1369rank 53–146 | $0.5 | $2 | 129K | Alibaba Released 2025-09 Price $0.5 / $2 Length 129K | ||||
| 38 | Llama 4 Scout | Meta | Rating 37.2 | 1259rank 149–209 | — | — | 128K | Meta Released 2025-04 Price — Length 128K | ||||
| 39 | ELYZA-Shortcut-1.0-Qwen-32B | ELYZA | Rating 34.6 | — | — | — | — | — | — | ELYZA Released — Price — Length — | ||
| 40 | Gemma 4 E2B | Rating 34.3 | — | — | — | 131.1K | — | — | Google Released — Price — Length 131.1K | |||
| 41 | ABEJA-QwQ32b-Reasoning-Japanese-v1.0 | ABEJA | Rating 34.0 | — | — | — | — | — | — | ABEJA Released — Price — Length — | ||
| 42 | Qwen3 30B A3B | Alibaba | Rating 33.4 | 1254rank 150–211 | — | — | 128K | Alibaba Released 2025-04 Price — Length 128K | ||||
| 43 | Qwen3 14B | Alibaba | Rating 32.3 | — | $0.35 | $1.4 | 131.1K | Alibaba Released 2025-04 Price $0.35 / $1.4 Length 131.1K | ||||
| 44 | GPT-4o (2024-08) | OpenAI | Rating 31.1 | 1292rank 124–185 | $2.5 | $10 | 128K | OpenAI Released 2024-08 Price $2.5 / $10 Length 128K | ||||
| 45 | Qwen3-Swallow-8B-CPT-v0.2 | Swallow (Science Tokyo) | Rating 30.8 | — | — | — | — | — | — | Swallow (Science Tokyo) Released — Price — Length — | ||
| 46–47 | Gemma 3 12B | Rating 29.4 | — | — | — | 131K | Google Released 2025-03 Price — Length 131K | |||||
| 46–47 | Qwen3 32B | Alibaba | Rating 29.4 | — | $0.7 | $2.8 | 131.1K | Alibaba Released 2025-04 Price $0.7 / $2.8 Length 131.1K | ||||
| 48 | Llama 3.1 8B (instruct) | Meta | Rating 29.1 | 1087rank 232–251 | — | — | 128K | Meta Released 2024-07 Price — Length 128K | ||||
| 49 | ELYZA-Thinking-1.0-Qwen-32B | ELYZA | Rating 28.8 | — | — | — | — | — | — | ELYZA Released — Price — Length — | ||
| 50 | Qwen3 8B | Alibaba | Rating 28.2 | — | $0.18 | $0.7 | 131.1K | Alibaba Released 2025-04 Price $0.18 / $0.7 Length 131.1K | ||||
| 51 | QwQ Bakeneko 32B | rinna | Rating 27.3 | — | — | — | — | — | — | rinna Released — Price — Length — | ||
| 52 | Qwen3 4B | Alibaba | Rating 27.0 | — | — | — | — | Alibaba Released 2025-04 Price — Length — | ||||
| 53 | DeepSeek-R1-Distill-Qwen-32B-Japanese | CyberAgent | Rating 26.7 | — | — | — | — | — | — | CyberAgent Released — Price — Length — | ||
| 54 | Gemma 3 27B | Rating 26.5 | 1308rank 117–173 | — | — | 128K | Google Released 2025-03 Price — Length 128K | |||||
| 55 | Llama 3.3 Swallow 70B (instruct, v0.4) | Swallow (Science Tokyo) | Rating 25.9 | — | — | — | — | — | — | Swallow (Science Tokyo) Released — Price — Length — | ||
| 56 | OLMo 3 32B Think | Ai2 | Rating 25.6 | — | — | — | — | — | Ai2 Released — Price — Length — | |||
| 57 | OLMo 3 7B Think | Ai2 | Rating 25.3 | — | — | — | — | — | — | Ai2 Released — Price — Length — | ||
| 58 | Gemma 3 4B | Rating 24.7 | — | — | — | 131K | Google Released 2025-03 Price — Length 131K | |||||
| 59 | Qwen3 1.7B | Alibaba | Rating 23.8 | — | — | — | — | Alibaba Released 2025-04 Price — Length — | ||||
| 60 | Llama 3.1 Nemotron Nano 8B (v1) | NVIDIA | Rating 23.5 | — | — | — | — | — | — | NVIDIA Released — Price — Length — | ||
| 61 | Qwen3.5 2B | Alibaba | Rating 22.1 | — | — | — | 32.8K | Alibaba Released 2026-02 Price — Length 32.8K | ||||
| 62 | Llama 3.1 Swallow 8B (instruct, v0.5) | Swallow (Science Tokyo) | Rating 20.6 | — | — | — | — | — | — | Swallow (Science Tokyo) Released — Price — Length — | ||
| 63 | Qwen3 0.6B | Alibaba | Rating 18.6 | — | — | — | — | Alibaba Released 2025-04 Price — Length — | ||||
| 64 | Sarashina2.2 3B (instruct, v0.1) | SB Intuitions | Rating 17.4 | — | — | — | — | — | — | SB Intuitions Released — Price — Length — | ||
| 65 | Qwen3.5 0.8B | Alibaba | Rating 16.6 | — | — | — | 32.8K | Alibaba Released 2026-02 Price — Length 32.8K | ||||
| 66 | LLM-jp-3.1 1.8B (instruct4) | LLM-jp | Rating 13.7 | — | — | — | — | — | — | LLM-jp Released — Price — Length — | ||
| 67 | Gemma 3 1B | Rating 13.4 | — | — | — | — | Google Released 2025-03 Price — Length — | |||||
| 68 | LLM-jp-3.1 13B (instruct4) | LLM-jp | Rating 13.1 | — | — | — | — | — | — | LLM-jp Released — Price — Length — |
How to read this page
- The value is the share followed. The use case pages use the values measured with the Japanese task set.
- The human rating comes from people comparing two answers and picking the better one (higher is better).
- The success rate is the share of tries that succeeded. A gap of 1 point is 1 success out of 100 tries.
- The same task measured with the Japanese tasks: Following format instructions given in Japanese (MIFEvalJa)
- In the Supports column the icons are, from left: can use tools / can read images / can answer in a fixed format (JSON); faded means not supported or unknown.
- Open-weights models have their contents (the weights) published, so you can also run them on your own computer or server. Models without the open-weights mark are normally used through the developer's own services (apps or API); for some models the weights status is unknown.
- Names such as ChatGPT next to a model are the consumer services its developer offers. Check on the service itself whether that model is available there.
Data: Arena, Epoch AI, Swallow LLM team (CC BY 4.0), Models.dev (MIT), updated 2026-10-24, data sources