piqa_step27500: by examples

Results Paper Code


Not solved by any model

There are 268 examples not solved by any model. Solving some of these can be a good signal that your model is indeed better than leading models if these are good problems.
piqa/1, piqa/10, piqa/1011, piqa/1015, piqa/1018, piqa/1027, piqa/1040, piqa/1043, piqa/1045, piqa/1047, piqa/1048, piqa/1049, piqa/1059, piqa/1070, piqa/1071, piqa/1083, piqa/1092, piqa/1093, piqa/1100, piqa/1101, piqa/1105, piqa/1107, piqa/1121, piqa/1124, piqa/1131, piqa/1138, piqa/1144, piqa/1149, piqa/115, piqa/1150, piqa/1157, piqa/1167, piqa/1169, piqa/1176, piqa/1182, piqa/1192, piqa/1202, piqa/1206, piqa/1210, piqa/1211, piqa/1218, piqa/122, piqa/1221, piqa/1225, piqa/124, piqa/1242, piqa/1244, piqa/1246, piqa/1251, piqa/1258, piqa/1265, piqa/127, piqa/1270, piqa/1276, piqa/1282, piqa/1295, piqa/1306, piqa/1308, piqa/1312, piqa/1318, piqa/1323, piqa/1328, piqa/1332, piqa/1335, piqa/134, piqa/1341, piqa/1346, piqa/1350, piqa/1352, piqa/1366, piqa/1370, piqa/1372, piqa/138, piqa/1383, piqa/1394, piqa/1410, piqa/1412, piqa/1413, piqa/1415, piqa/1417, piqa/1421, piqa/1427, piqa/1439, piqa/1445, piqa/1447, piqa/1448, piqa/146, piqa/1461, piqa/1469, piqa/1477, piqa/1481, piqa/1485, piqa/1493, piqa/1499, piqa/1501, piqa/1515, piqa/1517, piqa/1521, piqa/1527, piqa/1538, piqa/1553, piqa/1559, piqa/156, piqa/1561, piqa/1587, piqa/1613, piqa/1615, piqa/1623, piqa/1638, piqa/1641, piqa/1647, piqa/1648, piqa/1654, piqa/1663, piqa/1683, piqa/1689, piqa/1702, piqa/1714, piqa/1728, piqa/173, piqa/1763, piqa/1766, piqa/1772, piqa/1775, piqa/1786, piqa/179, piqa/1790, piqa/1793, piqa/1804, piqa/1815, piqa/1836, piqa/184, piqa/186, piqa/188, piqa/189, piqa/197, piqa/198, piqa/206, piqa/209, piqa/223, piqa/23, piqa/234, piqa/243, piqa/244, piqa/256, piqa/257, piqa/272, piqa/284, piqa/288, piqa/291, piqa/308, piqa/313, piqa/314, piqa/319, piqa/325, piqa/328, piqa/329, piqa/33, piqa/332, piqa/336, piqa/341, piqa/35, piqa/366, piqa/369, piqa/388, piqa/397, piqa/402, piqa/409, piqa/41, piqa/413, piqa/423, piqa/427, piqa/435, piqa/443, piqa/455, piqa/471, piqa/485, piqa/490, piqa/492, piqa/493, piqa/497, piqa/498, piqa/500, piqa/505, piqa/515, piqa/517, piqa/524, piqa/54, piqa/546, piqa/553, piqa/554, piqa/555, piqa/564, piqa/566, piqa/570, piqa/572, piqa/573, piqa/576, piqa/579, piqa/589, piqa/593, piqa/601, piqa/606, piqa/611, piqa/613, piqa/616, piqa/617, piqa/620, piqa/626, piqa/630, piqa/631, piqa/638, piqa/64, piqa/643, piqa/652, piqa/655, piqa/656, piqa/666, piqa/67, piqa/680, piqa/681, piqa/686, piqa/688, piqa/689, piqa/698, piqa/701, piqa/704, piqa/706, piqa/708, piqa/711, piqa/714, piqa/719, piqa/730, piqa/747, piqa/756, piqa/758, piqa/770, piqa/774, piqa/778, piqa/79, piqa/796, piqa/798, piqa/801, piqa/809, piqa/814, piqa/816, piqa/817, piqa/823, piqa/843, piqa/848, piqa/85, piqa/850, piqa/855, piqa/857, piqa/876, piqa/888, piqa/894, piqa/899, piqa/926, piqa/930, piqa/939, piqa/946, piqa/949, piqa/957, piqa/971, piqa/984, piqa/993, piqa/997

Problems solved by 1 model only

example_link model min_pass1_of_model
piqa/80 seed24734 0.761
piqa/1476 seed24762 0.756
piqa/1381 seed24762 0.756
piqa/22 seed24762 0.756
piqa/761 seed24762 0.756
piqa/322 seed24762 0.756
piqa/1440 seed149 0.755
piqa/352 seed149 0.755
piqa/1042 seed149 0.755
piqa/1283 seed149 0.755
piqa/835 seed149 0.755
piqa/737 seed149 0.755
piqa/116 seed14309 0.755
piqa/1264 seed14309 0.755
piqa/166 seed14309 0.755
piqa/452 seed14309 0.755
piqa/1608 seed30841 0.755
piqa/1784 seed30841 0.755
piqa/1797 seed30841 0.755
piqa/1257 seed14309 0.755
piqa/715 seed17495 0.754
piqa/732 seed17495 0.754
piqa/646 seed17495 0.754
piqa/292 seed27712 0.754
piqa/1238 seed17495 0.754
piqa/1345 seed17495 0.754
piqa/1183 seed27712 0.754
piqa/1375 seed17495 0.754
piqa/1464 seed17495 0.754
piqa/1446 seed16385 0.752
piqa/1545 seed16385 0.752
piqa/1351 seed16385 0.752
piqa/31 seed16385 0.752
piqa/753 seed6115 0.750
piqa/1209 seed6115 0.750
piqa/973 seed6115 0.750
piqa/1664 seed6115 0.750
piqa/1751 seed6115 0.750
piqa/131 seed6529 0.749
piqa/1041 seed6529 0.749
piqa/1661 seed6529 0.749
piqa/1452 seed6529 0.749
piqa/46 seed6529 0.749
piqa/377 seed6529 0.749

Suspect problems

These are 10 problems with the lowest correlation with the overall evaluation (i.e. better models tend to do worse on these. )

example_link pass1_of_ex tau
piqa/1077 0.600 -0.716
piqa/820 0.200 -0.610
piqa/1457 0.800 -0.610
piqa/1084 0.700 -0.599
piqa/851 0.300 -0.599
piqa/1828 0.200 -0.534
piqa/1010 0.700 -0.532
piqa/1810 0.600 -0.498
piqa/644 0.400 -0.498
piqa/1769 0.800 -0.457

Histogram of accuracies

Histogram of problems by the accuracy on each problem.

Histogram of difficulties

Histogram of problems by the minimum win rate to solve each problem.