There are 24 examples not solved by any model.
Solving some of these can be a good signal that your model is indeed better than leading models if these are good problems.
atcoder.abc373_g, atcoder.abc376_f, atcoder.abc389_g, atcoder.abc397_d, atcoder.arc181_c, atcoder.arc181_d, atcoder.arc182_d, atcoder.arc183_b, atcoder.arc183_d, atcoder.arc184_d, atcoder.arc186_c, atcoder.arc186_e, atcoder.arc188_c, atcoder.arc189_a, atcoder.arc190_a, atcoder.arc190_c, atcoder.arc191_c, atcoder.arc192_b, atcoder.arc193_b, atcoder.arc194_c, atcoder.arc196_a, atcoder.arc196_c, leetcode.3527, leetcode.3763
| example_link | model | min_pass1_of_model |
|---|---|---|
| atcoder.arc184_c | O4-Mini (High) | 0.802 |
| atcoder.arc192_e | O3 (High) | 0.758 |
| leetcode.3762 | O3 (High) | 0.758 |
| leetcode.3701 | O3 (High) | 0.758 |
| leetcode.3638 | O3 (High) | 0.758 |
| atcoder.arc183_c | O3 (High) | 0.758 |
| atcoder.arc191_a | DeepSeek-R1-0528 | 0.731 |
| atcoder.abc392_d | DeepSeek-R1-0528 | 0.731 |
| atcoder.abc374_d | DeepSeek-R1-0528 | 0.731 |
| atcoder.arc196_d | DeepSeek-R1-0528 | 0.731 |
| atcoder.abc400_g | DeepSeek-R1-0528 | 0.731 |
| atcoder.arc186_a | EXAONE-4.0-32B | 0.700 |
| atcoder.arc195_d | Gemini-2.5-Pro-03-25 | 0.678 |
| atcoder.abc370_g | Grok-3-Mini (High) | 0.667 |
| atcoder.abc371_f | Grok-3-Mini (High) | 0.667 |
| atcoder.abc366_g | Grok-3-Mini (High) | 0.667 |
| atcoder.abc375_b | Grok-3-Mini (High) | 0.667 |
| atcoder.abc382_g | Grok-3-Mini (High) | 0.667 |
| atcoder.abc378_g | Grok-3-Mini (High) | 0.667 |
| atcoder.abc376_g | Grok-3-Mini (High) | 0.667 |
| atcoder.abc399_e | Grok-3-Mini (High) | 0.667 |
| atcoder.arc185_c | Grok-3-Mini (High) | 0.667 |
| atcoder.abc398_g | Grok-3-Mini (High) | 0.667 |
| atcoder.arc195_c | Grok-3-Mini (High) | 0.667 |
| leetcode.3613 | Gemini-Flash-2.0-Thinking-12-19 | 0.487 |
These are 10 problems with the lowest correlation with the overall evaluation (i.e. better models tend to do worse on these. )
| example_link | pass1_of_ex | tau |
|---|---|---|
| atcoder.abc384_f | 0.756 | -0.273 |
| atcoder.abc367_b | 0.991 | -0.139 |
| atcoder.abc384_g | 0.598 | -0.137 |
| leetcode.3702 | 0.956 | -0.115 |
| leetcode.3613 | 0.030 | -0.108 |
| atcoder.abc375_a | 0.970 | -0.092 |
| atcoder.abc389_f | 0.547 | -0.071 |
| atcoder.abc386_f | 0.773 | -0.065 |
| atcoder.arc193_a | 0.271 | -0.063 |
| atcoder.abc378_a | 0.939 | 0.044 |
Histogram of problems by the accuracy on each problem.
Histogram of problems by the minimum win rate to solve each problem.