There are 268 examples not solved by any model.
Solving some of these can be a good signal that your model is indeed better than leading models if these are good problems.
piqa/1, piqa/10, piqa/1011, piqa/1015, piqa/1018, piqa/1027, piqa/1040, piqa/1043, piqa/1045, piqa/1047, piqa/1048, piqa/1049, piqa/1059, piqa/1070, piqa/1071, piqa/1083, piqa/1092, piqa/1093, piqa/1100, piqa/1101, piqa/1105, piqa/1107, piqa/1121, piqa/1124, piqa/1131, piqa/1138, piqa/1144, piqa/1149, piqa/115, piqa/1150, piqa/1157, piqa/1167, piqa/1169, piqa/1176, piqa/1182, piqa/1192, piqa/1202, piqa/1206, piqa/1210, piqa/1211, piqa/1218, piqa/122, piqa/1221, piqa/1225, piqa/124, piqa/1242, piqa/1244, piqa/1246, piqa/1251, piqa/1258, piqa/1265, piqa/127, piqa/1270, piqa/1276, piqa/1282, piqa/1295, piqa/1306, piqa/1308, piqa/1312, piqa/1318, piqa/1323, piqa/1328, piqa/1332, piqa/1335, piqa/134, piqa/1341, piqa/1346, piqa/1350, piqa/1352, piqa/1366, piqa/1370, piqa/1372, piqa/138, piqa/1383, piqa/1394, piqa/1410, piqa/1412, piqa/1413, piqa/1415, piqa/1417, piqa/1421, piqa/1427, piqa/1439, piqa/1445, piqa/1447, piqa/1448, piqa/146, piqa/1461, piqa/1469, piqa/1477, piqa/1481, piqa/1485, piqa/1493, piqa/1499, piqa/1501, piqa/1515, piqa/1517, piqa/1521, piqa/1527, piqa/1538, piqa/1553, piqa/1559, piqa/156, piqa/1561, piqa/1587, piqa/1613, piqa/1615, piqa/1623, piqa/1638, piqa/1641, piqa/1647, piqa/1648, piqa/1654, piqa/1663, piqa/1683, piqa/1689, piqa/1702, piqa/1714, piqa/1728, piqa/173, piqa/1763, piqa/1766, piqa/1772, piqa/1775, piqa/1786, piqa/179, piqa/1790, piqa/1793, piqa/1804, piqa/1815, piqa/1836, piqa/184, piqa/186, piqa/188, piqa/189, piqa/197, piqa/198, piqa/206, piqa/209, piqa/223, piqa/23, piqa/234, piqa/243, piqa/244, piqa/256, piqa/257, piqa/272, piqa/284, piqa/288, piqa/291, piqa/308, piqa/313, piqa/314, piqa/319, piqa/325, piqa/328, piqa/329, piqa/33, piqa/332, piqa/336, piqa/341, piqa/35, piqa/366, piqa/369, piqa/388, piqa/397, piqa/402, piqa/409, piqa/41, piqa/413, piqa/423, piqa/427, piqa/435, piqa/443, piqa/455, piqa/471, piqa/485, piqa/490, piqa/492, piqa/493, piqa/497, piqa/498, piqa/500, piqa/505, piqa/515, piqa/517, piqa/524, piqa/54, piqa/546, piqa/553, piqa/554, piqa/555, piqa/564, piqa/566, piqa/570, piqa/572, piqa/573, piqa/576, piqa/579, piqa/589, piqa/593, piqa/601, piqa/606, piqa/611, piqa/613, piqa/616, piqa/617, piqa/620, piqa/626, piqa/630, piqa/631, piqa/638, piqa/64, piqa/643, piqa/652, piqa/655, piqa/656, piqa/666, piqa/67, piqa/680, piqa/681, piqa/686, piqa/688, piqa/689, piqa/698, piqa/701, piqa/704, piqa/706, piqa/708, piqa/711, piqa/714, piqa/719, piqa/730, piqa/747, piqa/756, piqa/758, piqa/770, piqa/774, piqa/778, piqa/79, piqa/796, piqa/798, piqa/801, piqa/809, piqa/814, piqa/816, piqa/817, piqa/823, piqa/843, piqa/848, piqa/85, piqa/850, piqa/855, piqa/857, piqa/876, piqa/888, piqa/894, piqa/899, piqa/926, piqa/930, piqa/939, piqa/946, piqa/949, piqa/957, piqa/971, piqa/984, piqa/993, piqa/997
| example_link | model | min_pass1_of_model |
|---|---|---|
| piqa/80 | seed24734 | 0.761 |
| piqa/1476 | seed24762 | 0.756 |
| piqa/1381 | seed24762 | 0.756 |
| piqa/22 | seed24762 | 0.756 |
| piqa/761 | seed24762 | 0.756 |
| piqa/322 | seed24762 | 0.756 |
| piqa/1440 | seed149 | 0.755 |
| piqa/352 | seed149 | 0.755 |
| piqa/1042 | seed149 | 0.755 |
| piqa/1283 | seed149 | 0.755 |
| piqa/835 | seed149 | 0.755 |
| piqa/737 | seed149 | 0.755 |
| piqa/116 | seed14309 | 0.755 |
| piqa/1264 | seed14309 | 0.755 |
| piqa/166 | seed14309 | 0.755 |
| piqa/452 | seed14309 | 0.755 |
| piqa/1608 | seed30841 | 0.755 |
| piqa/1784 | seed30841 | 0.755 |
| piqa/1797 | seed30841 | 0.755 |
| piqa/1257 | seed14309 | 0.755 |
| piqa/715 | seed17495 | 0.754 |
| piqa/732 | seed17495 | 0.754 |
| piqa/646 | seed17495 | 0.754 |
| piqa/292 | seed27712 | 0.754 |
| piqa/1238 | seed17495 | 0.754 |
| piqa/1345 | seed17495 | 0.754 |
| piqa/1183 | seed27712 | 0.754 |
| piqa/1375 | seed17495 | 0.754 |
| piqa/1464 | seed17495 | 0.754 |
| piqa/1446 | seed16385 | 0.752 |
| piqa/1545 | seed16385 | 0.752 |
| piqa/1351 | seed16385 | 0.752 |
| piqa/31 | seed16385 | 0.752 |
| piqa/753 | seed6115 | 0.750 |
| piqa/1209 | seed6115 | 0.750 |
| piqa/973 | seed6115 | 0.750 |
| piqa/1664 | seed6115 | 0.750 |
| piqa/1751 | seed6115 | 0.750 |
| piqa/131 | seed6529 | 0.749 |
| piqa/1041 | seed6529 | 0.749 |
| piqa/1661 | seed6529 | 0.749 |
| piqa/1452 | seed6529 | 0.749 |
| piqa/46 | seed6529 | 0.749 |
| piqa/377 | seed6529 | 0.749 |
These are 10 problems with the lowest correlation with the overall evaluation (i.e. better models tend to do worse on these. )
| example_link | pass1_of_ex | tau |
|---|---|---|
| piqa/1077 | 0.600 | -0.716 |
| piqa/820 | 0.200 | -0.610 |
| piqa/1457 | 0.800 | -0.610 |
| piqa/1084 | 0.700 | -0.599 |
| piqa/851 | 0.300 | -0.599 |
| piqa/1828 | 0.200 | -0.534 |
| piqa/1010 | 0.700 | -0.532 |
| piqa/1810 | 0.600 | -0.498 |
| piqa/644 | 0.400 | -0.498 |
| piqa/1769 | 0.800 | -0.457 |
Histogram of problems by the accuracy on each problem.
Histogram of problems by the minimum win rate to solve each problem.