There are 36 examples not solved by any model.
Solving some of these can be a good signal that your model is indeed better than leading models if these are good problems.
atcoder.abc311_c, atcoder.abc315_e, atcoder.abc319_c, atcoder.abc324_f, atcoder.abc333_e, atcoder.abc343_a, atcoder.abc343_e, atcoder.abc350_c, atcoder.abc350_e, atcoder.abc362_c, atcoder.abc363_f, atcoder.abc373_g, atcoder.abc376_f, atcoder.abc389_g, atcoder.abc397_d, atcoder.arc181_c, atcoder.arc181_d, atcoder.arc182_d, atcoder.arc183_b, atcoder.arc183_d, atcoder.arc184_d, atcoder.arc186_c, atcoder.arc186_e, atcoder.arc188_c, atcoder.arc189_a, atcoder.arc190_a, atcoder.arc190_c, atcoder.arc191_c, atcoder.arc192_b, atcoder.arc193_b, atcoder.arc194_c, atcoder.arc196_a, atcoder.arc196_c, leetcode.3478, leetcode.3527, leetcode.3763
| example_link | model | min_pass1_of_model |
|---|---|---|
| atcoder.arc184_c | O4-Mini (High) | 0.873 |
| leetcode.3701 | O3 (High) | 0.847 |
| atcoder.abc355_e | O3 (High) | 0.847 |
| atcoder.arc192_e | O3 (High) | 0.847 |
| leetcode.3762 | O3 (High) | 0.847 |
| atcoder.arc183_c | O3 (High) | 0.847 |
| leetcode.3638 | O3 (High) | 0.847 |
| atcoder.abc400_g | DeepSeek-R1-0528 | 0.844 |
| atcoder.arc191_a | DeepSeek-R1-0528 | 0.844 |
| atcoder.abc315_f | DeepSeek-R1-0528 | 0.844 |
| atcoder.abc314_e | DeepSeek-R1-0528 | 0.844 |
| atcoder.abc327_e | DeepSeek-R1-0528 | 0.844 |
| atcoder.abc338_f | DeepSeek-R1-0528 | 0.844 |
| atcoder.abc392_d | DeepSeek-R1-0528 | 0.844 |
| atcoder.abc374_d | DeepSeek-R1-0528 | 0.844 |
| atcoder.arc196_d | DeepSeek-R1-0528 | 0.844 |
| atcoder.arc195_d | Gemini-2.5-Pro-03-25 | 0.815 |
| atcoder.arc186_a | EXAONE-4.0-32B | 0.809 |
| atcoder.abc378_g | Grok-3-Mini (High) | 0.781 |
| atcoder.abc370_g | Grok-3-Mini (High) | 0.781 |
| atcoder.abc371_f | Grok-3-Mini (High) | 0.781 |
| atcoder.abc366_g | Grok-3-Mini (High) | 0.781 |
| atcoder.abc375_b | Grok-3-Mini (High) | 0.781 |
| atcoder.arc185_c | Grok-3-Mini (High) | 0.781 |
| atcoder.abc382_g | Grok-3-Mini (High) | 0.781 |
| atcoder.abc376_g | Grok-3-Mini (High) | 0.781 |
| atcoder.abc399_e | Grok-3-Mini (High) | 0.781 |
| atcoder.arc195_c | Grok-3-Mini (High) | 0.781 |
| atcoder.abc398_g | Grok-3-Mini (High) | 0.781 |
| atcoder.abc337_e | O3-Mini-2025-01-31 (High) | 0.777 |
| leetcode.3613 | Gemini-Flash-2.0-Thinking-12-19 | 0.565 |
These are 10 problems with the lowest correlation with the overall evaluation (i.e. better models tend to do worse on these. )
| example_link | pass1_of_ex | tau |
|---|---|---|
| atcoder.abc384_f | 0.756 | -0.278 |
| leetcode.2816 | 0.970 | -0.246 |
| atcoder.abc344_b | 0.873 | -0.181 |
| atcoder.abc384_g | 0.598 | -0.176 |
| atcoder.abc367_b | 0.991 | -0.138 |
| leetcode.3702 | 0.956 | -0.138 |
| leetcode.3613 | 0.030 | -0.123 |
| atcoder.abc389_f | 0.547 | -0.090 |
| atcoder.abc386_f | 0.773 | -0.089 |
| atcoder.abc375_a | 0.970 | -0.077 |
Histogram of problems by the accuracy on each problem.
Histogram of problems by the minimum win rate to solve each problem.