Task results
Writing Python code from Japanese descriptions (JHumanEval)
Gives function descriptions in Japanese, has the model write Python code, and checks whether it passes prepared tests.
How to read this table
- Higher rows are rated higher. A range like “1–2” means the gap is within the margin of error.
- The score is the share of a shared set of problems the model solved (%).
- A gap of 3 means 3 problems out of 100.
- Prices are a guide per million tokens, reading then writing.
- Length is how much text the model can read at once (tokens).
| Rank | Model | Developer | Accuracy (%) | Evaluated | Japanese (human rating) | Input $ /1M | Output $ /1M | Context | Supports | Weights | Released | Price and specs |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | GPT-5.4 | OpenAI | Rating 96.3 | 1480rank 2–50 | $2.5 | $15 | 1.05M | OpenAI Released 2026-03 Price $2.5 / $15 Length 1.05M | ||||
| 2 | Gemma 4 26B A4B | Rating 95.9 | — | $0.15 | $0.6 | 262.1K | Google Released 2026-04 Price $0.15 / $0.6 Length 262.1K | |||||
| 3 | Gemma 4 31B | Rating 95.2 | — | — | — | 262.1K | Google Released 2026-04 Price — Length 262.1K | |||||
| 4 | Qwen3 235B A22B (thinking) | Alibaba | Rating 94.8 | 1376rank 49–142 | — | — | 262.1K | Alibaba Released 2025-07 Price — Length 262.1K | ||||
| 5 | GPT-5 | OpenAI | Rating 94.6 | 1431rank 17–93 | $1.25 | $10 | 400K | OpenAI Released 2025-08 Price $1.25 / $10 Length 400K | ||||
| 6 | GPT-5 mini | OpenAI | Rating 94.4 | 1364rank 63–146 | $0.25 | $2 | 400K | OpenAI Released 2025-08 Price $0.25 / $2 Length 400K | ||||
| 7–8 | o3 | OpenAI | Rating 93.7 | 1424rank 22–95 | $2 | $8 | 200K | OpenAI Released 2025-04 Price $2 / $8 Length 200K | ||||
| 7–8 | Qwen3.5 122B A10B | Alibaba | Rating 93.7 | 1361rank 60–154 | $0.4 | $3.2 | 262.1K | Alibaba Released 2026-02 Price $0.4 / $3.2 Length 262.1K | ||||
| 9 | GPT-OSS-Swallow-120B-RL-v0.1 | Swallow (Science Tokyo) | Rating 93.4 | — | — | — | — | — | — | Swallow (Science Tokyo) Released — Price — Length — | ||
| 10 | GPT-OSS 20B | OpenAI | Rating 93.2 | 1256rank 140–216 | — | — | 131K | OpenAI Released 2025-08 Price — Length 131K | ||||
| 11–12 | NVIDIA Nemotron 3 Super 120B A12B | NVIDIA | Rating 93.0 | — | — | — | 262.1K | NVIDIA Released 2026-03 Price — Length 262.1K | ||||
| 11–12 | Qwen3.5 27B | Alibaba | Rating 93.0 | 1339rank 71–166 | $0.3 | $2.4 | 262.1K | Alibaba Released 2026-02 Price $0.3 / $2.4 Length 262.1K | ||||
| 13 | Qwen3-Swallow-32B-RL-v0.2 | Swallow (Science Tokyo) | Rating 92.9 | — | — | — | — | — | — | Swallow (Science Tokyo) Released — Price — Length — | ||
| 14 | o3-mini | OpenAI | Rating 92.8 | 1315rank 101–175 | $1.1 | $4.4 | 200K | OpenAI Released 2025-01 Price $1.1 / $4.4 Length 200K | ||||
| 15–16 | GPT-OSS 120B | OpenAI | Rating 92.7 | 1325rank 95–166 | — | — | 128K | OpenAI Released 2025-08 Price — Length 128K | ||||
| 15–16 | GPT-OSS-Swallow-120B-SFT-v0.1 | Swallow (Science Tokyo) | Rating 92.7 | — | — | — | — | — | — | Swallow (Science Tokyo) Released — Price — Length — | ||
| 17 | GPT-OSS-Swallow-20B-RL-v0.1 | Swallow (Science Tokyo) | Rating 92.4 | — | — | — | — | — | — | Swallow (Science Tokyo) Released — Price — Length — | ||
| 18–20 | OLMo 3 32B Think | Ai2 | Rating 92.1 | — | — | — | — | — | Ai2 Released — Price — Length — | |||
| 18–20 | Qwen3 14B | Alibaba | Rating 92.1 | — | $0.35 | $1.4 | 131.1K | Alibaba Released 2025-04 Price $0.35 / $1.4 Length 131.1K | ||||
| 18–20 | Qwen3.5 35B A3B | Alibaba | Rating 92.1 | 1315rank 93–183 | $0.25 | $2 | 262.1K | Alibaba Released 2026-02 Price $0.25 / $2 Length 262.1K | ||||
| 21 | Qwen3-Swallow-30B-A3B-RL-v0.2 | Swallow (Science Tokyo) | Rating 91.7 | — | — | — | — | — | — | Swallow (Science Tokyo) Released — Price — Length — | ||
| 22–23 | Qwen3 30B A3B | Alibaba | Rating 91.5 | 1254rank 150–211 | — | — | 128K | Alibaba Released 2025-04 Price — Length 128K | ||||
| 22–23 | Qwen3 32B | Alibaba | Rating 91.5 | — | $0.7 | $2.8 | 131.1K | Alibaba Released 2025-04 Price $0.7 / $2.8 Length 131.1K | ||||
| 24–25 | GPT-4.1 | OpenAI | Rating 91.1 | 1347rank 83–150 | $2 | $8 | 1.05M | OpenAI Released 2025-04 Price $2 / $8 Length 1.05M | ||||
| 24–25 | Qwen3-Swallow-32B-SFT-v0.2 | Swallow (Science Tokyo) | Rating 91.1 | — | — | — | — | — | — | Swallow (Science Tokyo) Released — Price — Length — | ||
| 26–28 | GPT-OSS-Swallow-20B-SFT-v0.1 | Swallow (Science Tokyo) | Rating 90.9 | — | — | — | — | — | — | Swallow (Science Tokyo) Released — Price — Length — | ||
| 26–28 | Qwen3.5 9B | Alibaba | Rating 90.9 | — | — | — | 262K | Alibaba Released 2026-02 Price — Length 262K | ||||
| 26–28 | Qwen3-Swallow-32B-CPT-v0.2 | Swallow (Science Tokyo) | Rating 90.9 | — | — | — | — | — | — | Swallow (Science Tokyo) Released — Price — Length — | ||
| 29 | Qwen3 Next 80B A3B (instruct) | Alibaba | Rating 90.5 | 1369rank 53–146 | $0.5 | $2 | 129K | Alibaba Released 2025-09 Price $0.5 / $2 Length 129K | ||||
| 30 | Qwen3 235B A22B (instruct) | Alibaba | Rating 90.0 | 1392rank 44–119 | — | — | 128K | Alibaba Released 2025-07 Price — Length 128K | ||||
| 31 | OLMo 3 7B Think | Ai2 | Rating 89.6 | — | — | — | — | — | — | Ai2 Released — Price — Length — | ||
| 32–33 | Qwen3-Swallow-30B-A3B-CPT-v0.2 | Swallow (Science Tokyo) | Rating 89.5 | — | — | — | — | — | — | Swallow (Science Tokyo) Released — Price — Length — | ||
| 32–33 | Qwen3-Swallow-8B-RL-v0.2 | Swallow (Science Tokyo) | Rating 89.5 | — | — | — | — | — | — | Swallow (Science Tokyo) Released — Price — Length — | ||
| 34 | Qwen3 8B | Alibaba | Rating 89.1 | — | $0.18 | $0.7 | 131.1K | Alibaba Released 2025-04 Price $0.18 / $0.7 Length 131.1K | ||||
| 35–36 | ABEJA-QwQ32b-Reasoning-Japanese-v1.0 | ABEJA | Rating 89.0 | — | — | — | — | — | — | ABEJA Released — Price — Length — | ||
| 35–36 | Qwen3-Swallow-30B-A3B-SFT-v0.2 | Swallow (Science Tokyo) | Rating 89.0 | — | — | — | — | — | — | Swallow (Science Tokyo) Released — Price — Length — | ||
| 37 | NVIDIA-Nemotron-Nano-9B-v2-Japanese | NVIDIA | Rating 88.8 | — | — | — | — | — | — | NVIDIA Released — Price — Length — | ||
| 38 | Gemma 4 E4B | Rating 88.2 | — | — | — | 131.1K | — | — | Google Released — Price — Length 131.1K | |||
| 39 | Qwen3 4B | Alibaba | Rating 87.9 | — | — | — | — | Alibaba Released 2025-04 Price — Length — | ||||
| 40 | QwQ Bakeneko 32B | rinna | Rating 87.4 | — | — | — | — | — | — | rinna Released — Price — Length — | ||
| 41 | Qwen3 Next 80B A3B | Alibaba | Rating 87.3 | 1293rank 112–202 | $0.5 | $6 | 131.1K | Alibaba Released 2025-09 Price $0.5 / $6 Length 131.1K | ||||
| 42 | LLM-jp-4 32B A3B | LLM-jp | Rating 87.1 | — | — | — | — | — | — | LLM-jp Released — Price — Length — | ||
| 43 | GPT-4o (2024-08) | OpenAI | Rating 84.4 | 1292rank 124–185 | $2.5 | $10 | 128K | OpenAI Released 2024-08 Price $2.5 / $10 Length 128K | ||||
| 44 | LLM-jp-4 8B | LLM-jp | Rating 84.0 | — | — | — | — | — | — | LLM-jp Released — Price — Length — | ||
| 45 | Qwen3-Swallow-8B-SFT-v0.2 | Swallow (Science Tokyo) | Rating 82.9 | — | — | — | — | — | — | Swallow (Science Tokyo) Released — Price — Length — | ||
| 46 | Gemma 4 E2B | Rating 82.6 | — | — | — | 131.1K | — | — | Google Released — Price — Length 131.1K | |||
| 47 | Qwen3-Swallow-8B-CPT-v0.2 | Swallow (Science Tokyo) | Rating 82.3 | — | — | — | — | — | — | Swallow (Science Tokyo) Released — Price — Length — | ||
| 48 | Llama 4 Scout | Meta | Rating 82.0 | 1259rank 149–209 | — | — | 128K | Meta Released 2025-04 Price — Length 128K | ||||
| 49 | DeepSeek-R1-Distill-Qwen-32B-Japanese | CyberAgent | Rating 81.8 | — | — | — | — | — | — | CyberAgent Released — Price — Length — | ||
| 50 | ELYZA-Thinking-1.0-Qwen-32B | ELYZA | Rating 80.5 | — | — | — | — | — | — | ELYZA Released — Price — Length — | ||
| 51 | Llama 3.1 Nemotron Nano 8B (v1) | NVIDIA | Rating 80.4 | — | — | — | — | — | — | NVIDIA Released — Price — Length — | ||
| 52 | Gemma 3 27B | Rating 79.6 | 1308rank 117–173 | — | — | 128K | Google Released 2025-03 Price — Length 128K | |||||
| 53 | ELYZA-Shortcut-1.0-Qwen-32B | ELYZA | Rating 77.0 | — | — | — | — | — | — | ELYZA Released — Price — Length — | ||
| 54 | Gemma 3 12B | Rating 76.3 | — | — | — | 131K | Google Released 2025-03 Price — Length 131K | |||||
| 55 | Qwen3 1.7B | Alibaba | Rating 75.7 | — | — | — | — | Alibaba Released 2025-04 Price — Length — | ||||
| 56 | Llama 3.3 70B (instruct) | Meta | Rating 75.2 | 1214rank 176–224 | — | — | 131.1K | Meta Released 2024-12 Price — Length 131.1K | ||||
| 57–58 | Llama 3.3 Swallow 70B (instruct, v0.4) | Swallow (Science Tokyo) | Rating 72.7 | — | — | — | — | — | — | Swallow (Science Tokyo) Released — Price — Length — | ||
| 57–58 | Qwen3.5 4B | Alibaba | Rating 72.7 | — | — | — | 262.1K | Alibaba Released 2026-02 Price — Length 262.1K | ||||
| 59 | Gemma 3 4B | Rating 60.4 | — | — | — | 131K | Google Released 2025-03 Price — Length 131K | |||||
| 60 | Llama 3.1 Swallow 8B (instruct, v0.5) | Swallow (Science Tokyo) | Rating 58.4 | — | — | — | — | — | — | Swallow (Science Tokyo) Released — Price — Length — | ||
| 61 | Llama 3.1 8B (instruct) | Meta | Rating 58.0 | 1087rank 232–251 | — | — | 128K | Meta Released 2024-07 Price — Length 128K | ||||
| 62 | Sarashina2.2 3B (instruct, v0.1) | SB Intuitions | Rating 46.4 | — | — | — | — | — | — | SB Intuitions Released — Price — Length — | ||
| 63 | LLM-jp-3.1 13B (instruct4) | LLM-jp | Rating 46.3 | — | — | — | — | — | — | LLM-jp Released — Price — Length — | ||
| 64 | Qwen3 0.6B | Alibaba | Rating 44.6 | — | — | — | — | Alibaba Released 2025-04 Price — Length — | ||||
| 65 | LLM-jp-3.1 1.8B (instruct4) | LLM-jp | Rating 36.5 | — | — | — | — | — | — | LLM-jp Released — Price — Length — | ||
| 66 | Gemma 3 1B | Rating 11.2 | — | — | — | — | Google Released 2025-03 Price — Length — | |||||
| 67 | Qwen3.5 0.8B | Alibaba | Rating 1.3 | — | — | — | 32.8K | Alibaba Released 2026-02 Price — Length 32.8K | ||||
| 68 | Qwen3.5 2B | Alibaba | Rating 1.2 | — | — | — | 32.8K | Alibaba Released 2026-02 Price — Length 32.8K |
How to read this page
- The value is the share that pass.
- The human rating comes from people comparing two answers and picking the better one (higher is better).
- The accuracy is the share of problems solved. A gap of 1 point is 1 question out of 100.
- In the Supports column the icons are, from left: can use tools / can read images / can answer in a fixed format (JSON); faded means not supported or unknown.
- Open-weights models have their contents (the weights) published, so you can also run them on your own computer or server. Models without the open-weights mark are normally used through the developer's own services (apps or API); for some models the weights status is unknown.
- Names such as ChatGPT next to a model are the consumer services its developer offers. Check on the service itself whether that model is available there.
Data: Arena, Epoch AI, Swallow LLM team (CC BY 4.0), Models.dev (MIT), updated 2026-10-24, data sources