There are 408 examples not solved by any model.
Solving some of these can be a good signal that your model is indeed better than leading models if these are good problems.
piqa/1, piqa/10, piqa/1008, piqa/1011, piqa/1015, piqa/1025, piqa/1027, piqa/1039, piqa/1040, piqa/1041, piqa/1043, piqa/1045, piqa/1047, piqa/1048, piqa/1049, piqa/1050, piqa/1059, piqa/1060, piqa/1070, piqa/1071, piqa/1077, piqa/1083, piqa/1092, piqa/1093, piqa/110, piqa/1100, piqa/1101, piqa/1104, piqa/1107, piqa/1111, piqa/1121, piqa/1124, piqa/1131, piqa/1132, piqa/1138, piqa/1147, piqa/1148, piqa/1149, piqa/1150, piqa/1153, piqa/1157, piqa/116, piqa/1164, piqa/1167, piqa/1176, piqa/118, piqa/1182, piqa/1185, piqa/1186, piqa/1190, piqa/1192, piqa/1202, piqa/1206, piqa/1210, piqa/1211, piqa/122, piqa/1221, piqa/1223, piqa/1234, piqa/1239, piqa/1240, piqa/1242, piqa/1245, piqa/1246, piqa/1247, piqa/1251, piqa/1253, piqa/1258, piqa/1259, piqa/1264, piqa/1268, piqa/127, piqa/1270, piqa/1281, piqa/1282, piqa/1296, piqa/1306, piqa/1308, piqa/1312, piqa/1318, piqa/1319, piqa/1322, piqa/1325, piqa/1328, piqa/1330, piqa/1332, piqa/1335, piqa/134, piqa/1341, piqa/1344, piqa/1345, piqa/1346, piqa/1350, piqa/1351, piqa/1352, piqa/1360, piqa/1370, piqa/1371, piqa/1372, piqa/1375, piqa/1381, piqa/1383, piqa/1394, piqa/1398, piqa/1399, piqa/14, piqa/1400, piqa/1410, piqa/1411, piqa/1412, piqa/1413, piqa/1415, piqa/1417, piqa/1421, piqa/1423, piqa/1424, piqa/1427, piqa/1434, piqa/1439, piqa/1440, piqa/1445, piqa/1446, piqa/1448, piqa/1458, piqa/146, piqa/1461, piqa/1469, piqa/1473, piqa/1481, piqa/1485, piqa/1493, piqa/1501, piqa/1515, piqa/1521, piqa/153, piqa/1533, piqa/1538, piqa/1559, piqa/156, piqa/1561, piqa/1582, piqa/1585, piqa/1587, piqa/1590, piqa/160, piqa/1600, piqa/1612, piqa/1613, piqa/1615, piqa/1623, piqa/1630, piqa/1632, piqa/1638, piqa/1641, piqa/1647, piqa/1648, piqa/1650, piqa/1652, piqa/1654, piqa/1656, piqa/1659, piqa/1663, piqa/1664, piqa/1673, piqa/1683, piqa/1689, piqa/1694, piqa/1695, piqa/1701, piqa/1702, piqa/1714, piqa/172, piqa/1721, piqa/1722, piqa/1726, piqa/173, piqa/1746, piqa/1754, piqa/1762, piqa/1763, piqa/1766, piqa/1775, piqa/1781, piqa/1784, piqa/1786, piqa/179, piqa/1790, piqa/1793, piqa/1798, piqa/1799, piqa/1802, piqa/1804, piqa/1810, piqa/1820, piqa/1825, piqa/1827, piqa/1828, piqa/1836, piqa/184, piqa/186, piqa/188, piqa/189, piqa/197, piqa/2, piqa/206, piqa/207, piqa/209, piqa/221, piqa/223, piqa/23, piqa/24, piqa/244, piqa/25, piqa/255, piqa/256, piqa/257, piqa/266, piqa/267, piqa/268, piqa/269, piqa/272, piqa/28, piqa/280, piqa/281, piqa/284, piqa/287, piqa/288, piqa/292, piqa/308, piqa/309, piqa/312, piqa/313, piqa/314, piqa/315, piqa/322, piqa/324, piqa/325, piqa/326, piqa/328, piqa/329, piqa/33, piqa/332, piqa/335, piqa/336, piqa/341, piqa/35, piqa/352, piqa/359, piqa/365, piqa/366, piqa/379, piqa/388, piqa/394, piqa/398, piqa/402, piqa/409, piqa/41, piqa/411, piqa/413, piqa/415, piqa/423, piqa/440, piqa/443, piqa/447, piqa/453, piqa/455, piqa/456, piqa/46, piqa/461, piqa/462, piqa/47, piqa/471, piqa/485, piqa/490, piqa/492, piqa/493, piqa/496, piqa/497, piqa/498, piqa/500, piqa/502, piqa/505, piqa/508, piqa/517, piqa/52, piqa/533, piqa/539, piqa/54, piqa/544, piqa/546, piqa/554, piqa/564, piqa/565, piqa/566, piqa/570, piqa/571, piqa/572, piqa/573, piqa/576, piqa/579, piqa/589, piqa/601, piqa/606, piqa/609, piqa/611, piqa/613, piqa/616, piqa/617, piqa/620, piqa/623, piqa/626, piqa/630, piqa/631, piqa/637, piqa/638, piqa/64, piqa/643, piqa/644, piqa/652, piqa/655, piqa/656, piqa/666, piqa/67, piqa/679, piqa/68, piqa/680, piqa/681, piqa/686, piqa/688, piqa/689, piqa/694, piqa/695, piqa/698, piqa/701, piqa/704, piqa/705, piqa/706, piqa/708, piqa/71, piqa/711, piqa/714, piqa/719, piqa/721, piqa/725, piqa/730, piqa/732, piqa/735, piqa/737, piqa/747, piqa/753, piqa/756, piqa/761, piqa/762, piqa/770, piqa/773, piqa/774, piqa/778, piqa/779, piqa/784, piqa/789, piqa/79, piqa/796, piqa/799, piqa/80, piqa/801, piqa/814, piqa/817, piqa/820, piqa/823, piqa/827, piqa/832, piqa/835, piqa/843, piqa/848, piqa/85, piqa/855, piqa/857, piqa/862, piqa/863, piqa/870, piqa/871, piqa/881, piqa/888, piqa/892, piqa/894, piqa/899, piqa/904, piqa/905, piqa/924, piqa/926, piqa/929, piqa/930, piqa/934, piqa/939, piqa/943, piqa/944, piqa/946, piqa/949, piqa/95, piqa/955, piqa/957, piqa/971, piqa/978, piqa/984, piqa/990, piqa/992, piqa/993, piqa/997
| example_link | model | min_pass1_of_model |
|---|---|---|
| piqa/1003 | seed17495 | 0.650 |
| piqa/1009 | seed17495 | 0.650 |
| piqa/1036 | seed17495 | 0.650 |
| piqa/1119 | seed17495 | 0.650 |
| piqa/1379 | seed17495 | 0.650 |
| piqa/1366 | seed17495 | 0.650 |
| piqa/1357 | seed17495 | 0.650 |
| piqa/838 | seed17495 | 0.650 |
| piqa/593 | seed17495 | 0.650 |
| piqa/198 | seed17495 | 0.650 |
| piqa/317 | seed17495 | 0.650 |
| piqa/812 | seed17495 | 0.650 |
| piqa/1297 | seed17495 | 0.650 |
| piqa/1317 | seed17495 | 0.650 |
| piqa/515 | seed6529 | 0.649 |
| piqa/177 | seed6529 | 0.649 |
| piqa/1123 | seed6529 | 0.649 |
| piqa/586 | seed6529 | 0.649 |
| piqa/1508 | seed6529 | 0.649 |
| piqa/1608 | seed6529 | 0.649 |
| piqa/809 | seed6529 | 0.649 |
| piqa/549 | seed6529 | 0.649 |
| piqa/937 | seed24762 | 0.645 |
| piqa/1465 | seed24762 | 0.645 |
| piqa/1765 | seed24762 | 0.645 |
| piqa/397 | seed24762 | 0.645 |
| piqa/1571 | seed24762 | 0.645 |
| piqa/1225 | seed24762 | 0.645 |
| piqa/1311 | seed16385 | 0.645 |
| piqa/1758 | seed16385 | 0.645 |
| piqa/1429 | seed16385 | 0.645 |
| piqa/42 | seed16385 | 0.645 |
| piqa/427 | seed16385 | 0.645 |
| piqa/98 | seed16385 | 0.645 |
| piqa/1606 | seed16385 | 0.645 |
| piqa/966 | seed30841 | 0.644 |
| piqa/587 | seed30841 | 0.644 |
| piqa/1283 | seed30841 | 0.644 |
| piqa/1356 | seed30841 | 0.644 |
| piqa/1542 | seed30841 | 0.644 |
| piqa/1447 | seed30841 | 0.644 |
| piqa/1034 | seed30841 | 0.644 |
| piqa/1169 | seed24734 | 0.642 |
| piqa/1467 | seed24734 | 0.642 |
| piqa/1374 | seed24734 | 0.642 |
| piqa/138 | seed24734 | 0.642 |
| piqa/1482 | seed24734 | 0.642 |
| piqa/1661 | seed24734 | 0.642 |
| piqa/107 | seed27712 | 0.640 |
| piqa/952 | seed27712 | 0.640 |
| piqa/31 | seed149 | 0.640 |
| piqa/351 | seed149 | 0.640 |
| piqa/1244 | seed149 | 0.640 |
| piqa/1078 | seed149 | 0.640 |
| piqa/1235 | seed27712 | 0.640 |
| piqa/1183 | seed27712 | 0.640 |
| piqa/115 | seed149 | 0.640 |
| piqa/1266 | seed149 | 0.640 |
| piqa/131 | seed149 | 0.640 |
| piqa/1432 | seed149 | 0.640 |
| piqa/847 | seed149 | 0.640 |
| piqa/369 | seed149 | 0.640 |
| piqa/1545 | seed27712 | 0.640 |
| piqa/1499 | seed149 | 0.640 |
| piqa/1463 | seed149 | 0.640 |
| piqa/1103 | seed6115 | 0.637 |
| piqa/1085 | seed14309 | 0.637 |
| piqa/1105 | seed6115 | 0.637 |
| piqa/1643 | seed6115 | 0.637 |
| piqa/1498 | seed6115 | 0.637 |
| piqa/1610 | seed6115 | 0.637 |
| piqa/1551 | seed6115 | 0.637 |
| piqa/1547 | seed6115 | 0.637 |
| piqa/553 | seed14309 | 0.637 |
| piqa/200 | seed14309 | 0.637 |
| piqa/277 | seed14309 | 0.637 |
| piqa/717 | seed14309 | 0.637 |
| piqa/815 | seed14309 | 0.637 |
| piqa/726 | seed6115 | 0.637 |
| piqa/914 | seed14309 | 0.637 |
| piqa/907 | seed14309 | 0.637 |
These are 10 problems with the lowest correlation with the overall evaluation (i.e. better models tend to do worse on these. )
| example_link | pass1_of_ex | tau |
|---|---|---|
| piqa/1257 | 0.800 | -0.610 |
| piqa/1299 | 0.700 | -0.566 |
| piqa/1707 | 0.800 | -0.534 |
| piqa/755 | 0.600 | -0.498 |
| piqa/235 | 0.500 | -0.488 |
| piqa/1422 | 0.600 | -0.467 |
| piqa/662 | 0.900 | -0.457 |
| piqa/599 | 0.900 | -0.457 |
| piqa/673 | 0.900 | -0.457 |
| piqa/420 | 0.900 | -0.457 |
Histogram of problems by the accuracy on each problem.
Histogram of problems by the minimum win rate to solve each problem.