The typical standard errors between pairs of models on this dataset as a function of the absolute accuracy.
The standard error of each model pair against their observed accuracy difference. Pairs below a reference line differ by more than the corresponding number of standard errors, i.e. they are statistically distinguishable at that level.
| model | pass1 | pass@count | win_rate | count | SE(A) | SE_x(A) | SE_pred(A) |
|---|---|---|---|---|---|---|---|
| Kimi-k1.6-IOI-high | 86 | 86 | 27.8 | 1 | 1.2 | NaN | NaN |
| O1-2024-12-17 (High) | 83.2 | 83.2 | 26 | 1 | 1.3 | NaN | NaN |
| Kimi-k1.6-IOI | 80.1 | 88.5 | 23.4 | 10 | 1.3 | 1.2 | 0.69 |
| QwQ-Max-Preview | 80 | 80 | 23.2 | 1 | 1.3 | NaN | NaN |
| O1-2024-12-17 (Med) | 78.3 | 78.3 | 22.1 | 1 | 1.4 | NaN | NaN |
| DeepSeek-R1-Preview | 77.9 | 87 | 21.5 | 10 | 1.4 | 1.2 | 0.69 |
| Llama-3_1-Nemotron-Ultra-253B-v1 | 77.7 | 77.7 | 22.3 | 1 | 1.4 | NaN | NaN |
| O1-2024-12-17 (Low) | 75.9 | 75.9 | 20.7 | 1 | 1.4 | NaN | NaN |
| DeepCoder-14B-Preview | 73.3 | 73.3 | 20.6 | 1 | 1.5 | NaN | NaN |
| O1-Mini-2024-09-12 | 68.4 | 68.4 | 16.7 | 1 | 1.6 | NaN | NaN |
| Llama-3_1-Nemotron-Nano-8B-v1 | 64.4 | 64.4 | 15 | 1 | 1.6 | NaN | NaN |
| DeepSeek-R1-Lite-Preview | 63.1 | 79.5 | 13.3 | 10 | 1.6 | 1.4 | 0.89 |
| QwQ-32B-Preview | 59.9 | 59.9 | 11.5 | 1 | 1.7 | NaN | NaN |
| O1-Preview-2024-09-12 | 55.6 | 55.6 | 10.5 | 1 | 1.7 | NaN | NaN |
| DeepSeek-V3 copy | 54.5 | 63.6 | 9.84 | 10 | 1.7 | 1.6 | 0.61 |
| MetaStone-L1-7B | 54.1 | 74.3 | 10 | 10 | 1.7 | 1.4 | 0.96 |
| Gemini-Flash-2.0-Thinking | 51.1 | 64.1 | 8.08 | 10 | 1.7 | 1.5 | 0.77 |
| Claude-3.5-Sonnet-20240620 | 48 | 51.4 | 7.45 | 10 | 1.7 | 1.6 | 0.4 |
| GPT-4O-2024-05-13 | 43.4 | 50.7 | 5.25 | 10 | 1.7 | 1.6 | 0.53 |
| Gemini-Pro-1.5-002 | 42.1 | 47.7 | 4.95 | 10 | 1.7 | 1.6 | 0.47 |
| Mistral-Large | 37.3 | 47.4 | 3.66 | 10 | 1.6 | 1.5 | 0.62 |
| Gemini-Flash-1.5-002 | 36.1 | 39.2 | 3.36 | 10 | 1.6 | 1.6 | 0.34 |
| Codestral-Latest | 35.3 | 42.2 | 3.31 | 10 | 1.6 | 1.5 | 0.53 |
| AzeroGPT-64b | 22 | 40.7 | 1.15 | 10 | 1.4 | 1.1 | 0.86 |