The typical standard errors between pairs of models on this dataset as a function of the absolute accuracy.
The standard error of each model pair against their observed accuracy difference. Pairs below a reference line differ by more than the corresponding number of standard errors, i.e. they are statistically distinguishable at that level.
| model | pass1 | pass@count | win_rate | count | SE(A) | SE_x(A) | SE_pred(A) |
|---|---|---|---|---|---|---|---|
| Claude 4.5 Opus medium (20251101) | 74.4 | 74.4 | 24 | 1 | 2 | NaN | NaN |
| Gemini 3 Pro Preview (2025-11-18) | 74.2 | 74.2 | 23.8 | 1 | 2 | NaN | NaN |
| GPT-5.2 (2025-12-11) (high reasoning) | 71.8 | 71.8 | 22.6 | 1 | 2 | NaN | NaN |
| Claude 4.5 Sonnet (20250929) | 70.6 | 70.6 | 21.6 | 1 | 2 | NaN | NaN |
| GPT-5.2 (2025-12-11) | 69 | 69 | 21.3 | 1 | 2.1 | NaN | NaN |
| Claude 4 Opus (20250514) | 67.6 | 67.6 | 19.2 | 1 | 2.1 | NaN | NaN |
| GPT-5.1 (2025-11-13) (medium reasoning) | 66 | 66 | 18.8 | 1 | 2.1 | NaN | NaN |
| GPT-5.1-codex (medium reasoning) | 66 | 66 | 18.5 | 1 | 2.1 | NaN | NaN |
| GPT-5 (2025-08-07) (medium reasoning) | 65 | 65 | 17.9 | 1 | 2.1 | NaN | NaN |
| Claude 4 Sonnet (20250514) | 64.8 | 64.8 | 17.8 | 1 | 2.1 | NaN | NaN |
| Kimi K2 Thinking | 63.4 | 63.4 | 17.3 | 1 | 2.2 | NaN | NaN |
| Minimax M2 | 61 | 61 | 16.8 | 1 | 2.2 | NaN | NaN |
| DeepSeek V3.2 Reasoner | 60 | 60 | 16.8 | 1 | 2.2 | NaN | NaN |
| GPT-5 mini (2025-08-07) (medium reasoning) | 59.8 | 59.8 | 15.4 | 1 | 2.2 | NaN | NaN |
| o3 (2025-04-16) | 58.4 | 58.4 | 15.2 | 1 | 2.2 | NaN | NaN |
| Devstral small (2512) | 56.4 | 56.4 | 14.9 | 1 | 2.2 | NaN | NaN |
| Qwen3-Coder 480B/A35B Instruct | 55.4 | 55.4 | 14.3 | 1 | 2.2 | NaN | NaN |
| GLM-4.6 (T=1) | 55.4 | 55.4 | 13.8 | 1 | 2.2 | NaN | NaN |
| GLM-4.5 (2025-08-22) | 54.2 | 54.2 | 13 | 1 | 2.2 | NaN | NaN |
| Devstral (2512) | 53.8 | 53.8 | 14.2 | 1 | 2.2 | NaN | NaN |
| Gemini 2.5 Pro (2025-05-06) | 53.6 | 53.6 | 13 | 1 | 2.2 | NaN | NaN |
| o4-mini (2025-04-16) | 45 | 45 | 9.96 | 1 | 2.2 | NaN | NaN |
| Kimi K2 Instruct | 43.8 | 43.8 | 10.3 | 1 | 2.2 | NaN | NaN |
| GPT-5 nano (2025-08-07) (medium reasoning) | 34.8 | 34.8 | 7.36 | 1 | 2.1 | NaN | NaN |
| gpt-oss-120b | 26 | 26 | 5.24 | 1 | 2 | NaN | NaN |
| Llama 4 Maverick Instruct | 21 | 21 | 3.91 | 1 | 1.8 | NaN | NaN |
| Claude 3.7 Sonnet (20250219) | 10.2 | 10.2 | 2.13 | 1 | 1.4 | NaN | NaN |
| Qwen2.5-Coder 32B Instruct | 9 | 9 | 1.27 | 1 | 1.3 | NaN | NaN |