There are 267 examples not solved by any model.
Solving some of these can be a good signal that your model is indeed better than leading models if these are good problems.
piqa/1, piqa/10, piqa/1015, piqa/1018, piqa/1027, piqa/1040, piqa/1043, piqa/1045, piqa/1047, piqa/1049, piqa/1050, piqa/1059, piqa/1067, piqa/1071, piqa/1083, piqa/1092, piqa/1100, piqa/1101, piqa/1105, piqa/1107, piqa/1121, piqa/1124, piqa/1131, piqa/1138, piqa/1144, piqa/1149, piqa/115, piqa/1150, piqa/1157, piqa/1169, piqa/1182, piqa/1183, piqa/1185, piqa/1192, piqa/1202, piqa/1206, piqa/1209, piqa/1210, piqa/1211, piqa/1218, piqa/122, piqa/1221, piqa/1225, piqa/1238, piqa/1244, piqa/1246, piqa/1251, piqa/1257, piqa/1258, piqa/1261, piqa/1264, piqa/1265, piqa/127, piqa/1270, piqa/1282, piqa/1295, piqa/1306, piqa/1308, piqa/131, piqa/1312, piqa/1318, piqa/1319, piqa/1323, piqa/1332, piqa/134, piqa/1341, piqa/1344, piqa/1346, piqa/1350, piqa/1352, piqa/1366, piqa/1370, piqa/1372, piqa/1375, piqa/138, piqa/1381, piqa/1383, piqa/1394, piqa/1410, piqa/1412, piqa/1413, piqa/1415, piqa/1417, piqa/1421, piqa/1427, piqa/1439, piqa/1440, piqa/1445, piqa/1447, piqa/1448, piqa/146, piqa/1461, piqa/1469, piqa/1477, piqa/1481, piqa/1485, piqa/1493, piqa/1499, piqa/1501, piqa/1508, piqa/1515, piqa/1517, piqa/1521, piqa/1545, piqa/1553, piqa/156, piqa/1561, piqa/1587, piqa/161, piqa/1613, piqa/1615, piqa/1623, piqa/1638, piqa/1641, piqa/1647, piqa/1648, piqa/1654, piqa/1656, piqa/1660, piqa/1663, piqa/1683, piqa/1689, piqa/1702, piqa/1714, piqa/1728, piqa/173, piqa/1751, piqa/1766, piqa/1772, piqa/1775, piqa/1786, piqa/179, piqa/1790, piqa/1804, piqa/1815, piqa/1836, piqa/184, piqa/186, piqa/188, piqa/189, piqa/197, piqa/198, piqa/206, piqa/209, piqa/22, piqa/223, piqa/23, piqa/230, piqa/234, piqa/243, piqa/244, piqa/256, piqa/257, piqa/271, piqa/272, piqa/284, piqa/288, piqa/308, piqa/313, piqa/319, piqa/325, piqa/328, piqa/329, piqa/33, piqa/332, piqa/35, piqa/366, piqa/369, piqa/377, piqa/388, piqa/397, piqa/402, piqa/409, piqa/413, piqa/423, piqa/427, piqa/435, piqa/443, piqa/455, piqa/47, piqa/471, piqa/485, piqa/490, piqa/492, piqa/493, piqa/497, piqa/498, piqa/500, piqa/505, piqa/515, piqa/524, piqa/54, piqa/546, piqa/553, piqa/554, piqa/564, piqa/566, piqa/570, piqa/572, piqa/573, piqa/576, piqa/579, piqa/589, piqa/593, piqa/601, piqa/613, piqa/616, piqa/617, piqa/620, piqa/626, piqa/630, piqa/631, piqa/638, piqa/64, piqa/643, piqa/646, piqa/652, piqa/655, piqa/656, piqa/666, piqa/67, piqa/680, piqa/681, piqa/688, piqa/689, piqa/698, piqa/706, piqa/708, piqa/71, piqa/711, piqa/714, piqa/719, piqa/730, piqa/747, piqa/756, piqa/758, piqa/762, piqa/770, piqa/774, piqa/778, piqa/79, piqa/796, piqa/798, piqa/809, piqa/814, piqa/816, piqa/817, piqa/823, piqa/848, piqa/85, piqa/850, piqa/855, piqa/857, piqa/876, piqa/888, piqa/894, piqa/899, piqa/930, piqa/939, piqa/946, piqa/949, piqa/957, piqa/971, piqa/973, piqa/984, piqa/993, piqa/997
| example_link | model | min_pass1_of_model |
|---|---|---|
| piqa/803 | seed30841 | 0.776 |
| piqa/926 | seed30841 | 0.776 |
| piqa/968 | seed30841 | 0.776 |
| piqa/980 | seed30841 | 0.776 |
| piqa/1538 | seed30841 | 0.776 |
| piqa/46 | seed30841 | 0.776 |
| piqa/1793 | seed16385 | 0.773 |
| piqa/606 | seed16385 | 0.773 |
| piqa/1464 | seed16385 | 0.773 |
| piqa/1527 | seed16385 | 0.773 |
| piqa/1763 | seed16385 | 0.773 |
| piqa/1351 | seed16385 | 0.773 |
| piqa/1070 | seed17495 | 0.773 |
| piqa/873 | seed17495 | 0.773 |
| piqa/453 | seed17495 | 0.773 |
| piqa/1498 | seed17495 | 0.773 |
| piqa/1335 | seed149 | 0.770 |
| piqa/352 | seed149 | 0.770 |
| piqa/686 | seed14309 | 0.769 |
| piqa/336 | seed14309 | 0.769 |
| piqa/842 | seed14309 | 0.769 |
| piqa/14 | seed14309 | 0.769 |
| piqa/1659 | seed6529 | 0.769 |
| piqa/517 | seed6529 | 0.769 |
| piqa/1048 | seed24762 | 0.768 |
| piqa/753 | seed24762 | 0.768 |
| piqa/95 | seed24762 | 0.768 |
| piqa/1176 | seed24762 | 0.768 |
| piqa/41 | seed24762 | 0.768 |
| piqa/611 | seed24734 | 0.768 |
| piqa/1242 | seed24734 | 0.768 |
| piqa/1476 | seed24734 | 0.768 |
| piqa/1132 | seed6115 | 0.766 |
| piqa/124 | seed6115 | 0.766 |
| piqa/1167 | seed27712 | 0.761 |
| piqa/292 | seed27712 | 0.761 |
| piqa/502 | seed27712 | 0.761 |
These are 10 problems with the lowest correlation with the overall evaluation (i.e. better models tend to do worse on these. )
| example_link | pass1_of_ex | tau |
|---|---|---|
| piqa/1642 | 0.800 | -0.596 |
| piqa/1313 | 0.700 | -0.553 |
| piqa/1463 | 0.800 | -0.522 |
| piqa/1247 | 0.800 | -0.522 |
| piqa/291 | 0.200 | -0.522 |
| piqa/1559 | 0.200 | -0.522 |
| piqa/1042 | 0.800 | -0.522 |
| piqa/1630 | 0.500 | -0.507 |
| piqa/582 | 0.400 | -0.487 |
| piqa/1348 | 0.900 | -0.447 |
Histogram of problems by the accuracy on each problem.
Histogram of problems by the minimum win rate to solve each problem.