Skip to main content
Bench

Language

English日本語中文Español

Site info

Task results

Fixing real Python bugs (SWE-Bench verified)

Tasks where the model investigates and fixes GitHub issues filed against public Python repositories, using tasks checked by human developers.

How to read this table

  • Higher rows are rated higher. A range like “1–2” means the gap is within the margin of error.
  • The score is the share of tries on a shared set of tasks that the model completed (%).
  • A gap of 3 means 3 successes in 100 tries; ± is how far the score moves when re-measured.
  • Prices are a guide per million tokens, reading then writing.
  • Length is how much text the model can read at once (tokens).

± range of the top model

RankModelDeveloperSuccess rate
(%)Success rate scale 0 to 100		EvaluatedInput $
/1M		Output $
/1M		ContextSupportsWeightsReleasedPrice and specs
1–5Claude Opus 4.7AnthropicRating 83.5 ±3.3$5$251MAnthropic Released 2026-⁠04 Price $5 / $25 Length 1M
1–13GPT-5.5OpenAIRating 80.6 ±3.5$5$301.05MOpenAI Released 2026-⁠04 Price $5 / $30 Length 1.05M
1–16Gemini 3.5 FlashGoogleRating 79.3 ±3.6$1.5$91.05MGoogle Released 2026-⁠05 Price $1.5 / $9 Length 1.05M
1–20Claude Opus 4.6AnthropicRating 78.7 ±3.6$5$251MAnthropic Released 2026-⁠02 Price $5 / $25 Length 1M
1–20GLM-5.2⁠Z.aiRating 78.7 ±3.7$1.4$4.41MZ.ai Released 2026-⁠06 Price $1.4 / $4.4 Length 1M
2–21DeepSeek-V4 Pro⁠DeepSeekRating 77.6 ±3.7$0.66$1.981MDeepSeek Released 2026-⁠04 Price $0.66 / $1.98 Length 1M
2–22Qwen3.7 MaxAlibabaRating 77.3 ±3.7$2.5$7.51MAlibaba Released 2026-⁠05 Price $2.5 / $7.5 Length 1M
2–22GPT-5.4OpenAIRating 76.9 ±3.8$2.5$151.05MOpenAI Released 2026-⁠03 Price $2.5 / $15 Length 1.05M
2–23Claude Opus 4.5AnthropicRating 76.7 ±3.8$5$25200KAnthropic Released 2025-⁠11 Price $5 / $25 Length 200K
2–23Kimi K2.6⁠MoonshotRating 76.7 ±3.8$0.95$4262.1KMoonshot Released 2026-⁠04 Price $0.95 / $4 Length 262.1K
2–23Qwen3.6 Max (preview)AlibabaRating 76.7 ±3.8$1.3$7.8240KAlibaba Released 2026-⁠04 Price $1.3 / $7.8 Length 240K
2–24Gemini 3.1 Pro (preview)GoogleRating 75.6 ±3.8$2$121.05MGoogle Released 2026-⁠02 Price $2 / $12 Length 1.05M
2–24Gemini 3 Flash (preview)GoogleRating 75.4 ±3.8$0.5$31MGoogle Released 2025-⁠12 Price $0.5 / $3 Length 1M
3–24Claude Sonnet 4.6AnthropicRating 75.2 ±3.9$3$151MAnthropic Released 2026-⁠02 Price $3 / $15 Length 1M
3–24GPT-5.3 CodexOpenAIRating 74.8 ±3.9$1.75$14400KOpenAI Released 2026-⁠02 Price $1.75 / $14 Length 400K
3–24GLM-5.1⁠Z.aiRating 74.2 ±3.9$1.4$4.4200KZ.ai Released 2026-⁠04 Price $1.4 / $4.4 Length 200K
4–24GPT-5.2OpenAIRating 73.8 ±3.9$1.75$14400KOpenAI Released 2025-⁠12 Price $1.75 / $14 Length 400K
4–24Kimi K2.5⁠MoonshotRating 73.8 ±3.9——262.1KMoonshot Released 2026-⁠01 Price — Length 262.1K
4–25GPT-5OpenAIRating 73.6 ±3.9$1.25$10400KOpenAI Released 2025-⁠08 Price $1.25 / $10 Length 400K
4–25Claude Opus 4.1AnthropicRating 73.3 ±3.9——200KAnthropic Released 2025-⁠08 Price — Length 200K
6–25Gemini 3 Pro (preview)GoogleRating 72.9 ±4.0——1MGoogle Released 2025-⁠11 Price — Length 1M
7–25GLM-5⁠Z.aiRating 72.1 ±4.1$1$3.2204.8KZ.ai Released 2026-⁠02 Price $1 / $3.2 Length 204.8K
9–25Claude Sonnet 4.5AnthropicRating 71.3 ±4.0$3$15200KAnthropic Released 2025-⁠09 Price $3 / $15 Length 200K
12–25Claude Opus 4AnthropicRating 70.7 ±4.1——200KAnthropic Released 2025-⁠05 Price — Length 200K
19–27GPT-5.1OpenAIRating 68.0 ±4.2$1.25$10400KOpenAI Released 2025-⁠11 Price $1.25 / $10 Length 400K
25–28GPT-5 miniOpenAIRating 64.7 ±4.3$0.25$2400KOpenAI Released 2025-⁠08 Price $0.25 / $2 Length 400K
25–30o3OpenAIRating 62.3 ±4.3$2$8200KOpenAI Released 2025-⁠04 Price $2 / $8 Length 200K
26–30Claude 3.7 SonnetAnthropicRating 61.0 ±4.4——200KAnthropic Released 2025-⁠02 Price — Length 200K
27–30Qwen3.6 PlusAlibabaRating 57.9 ±4.4$0.5$31MAlibaba Released 2026-⁠03 Price $0.5 / $3 Length 1M
27–30Gemini 2.5 ProGoogleRating 57.6 ±4.4$1.25$101MGoogle Released 2025-⁠06 Price $1.25 / $10 Length 1M
31GPT-4.1OpenAIRating 48.5 ±4.5$2$81.05MOpenAI Released 2025-⁠04 Price $2 / $8 Length 1.05M
32GPT-4o (2024-⁠11)OpenAIRating 31.0 ±4.1$2.5$10128KOpenAI Released 2024-⁠11 Price $2.5 / $10 Length 128K

Best success rate over time

2024-11-20Release date2026-06-16
How to read this page
  • The fixed code is graded by running the repository's unit tests, and the value is the share of tasks resolved.
  • The success rate is the share of tries that succeeded. A gap of 1 point is 1 success out of 100 tries.
  • In the Supports column the icons are, from left: can use tools / can read images / can answer in a fixed format (JSON); faded means not supported or unknown.
  • Open-weights models have their contents (the weights) published, so you can also run them on your own computer or server. Models without the open-weights mark are normally used through the developer's own services (apps or API); for some models the weights status is unknown.
  • Names such as ChatGPT next to a model are the consumer services its developer offers. Check on the service itself whether that model is available there.

Data: Arena, Epoch AI, Swallow LLM team (CC BY 4.0), Models.dev (MIT), updated 2026-⁠10-⁠24, data sources