piqa_step50000: by examples

Results Paper Code


Not solved by any model

There are 267 examples not solved by any model. Solving some of these can be a good signal that your model is indeed better than leading models if these are good problems.
piqa/1, piqa/10, piqa/1015, piqa/1018, piqa/1027, piqa/1040, piqa/1043, piqa/1045, piqa/1047, piqa/1049, piqa/1050, piqa/1059, piqa/1067, piqa/1071, piqa/1083, piqa/1092, piqa/1100, piqa/1101, piqa/1105, piqa/1107, piqa/1121, piqa/1124, piqa/1131, piqa/1138, piqa/1144, piqa/1149, piqa/115, piqa/1150, piqa/1157, piqa/1169, piqa/1182, piqa/1183, piqa/1185, piqa/1192, piqa/1202, piqa/1206, piqa/1209, piqa/1210, piqa/1211, piqa/1218, piqa/122, piqa/1221, piqa/1225, piqa/1238, piqa/1244, piqa/1246, piqa/1251, piqa/1257, piqa/1258, piqa/1261, piqa/1264, piqa/1265, piqa/127, piqa/1270, piqa/1282, piqa/1295, piqa/1306, piqa/1308, piqa/131, piqa/1312, piqa/1318, piqa/1319, piqa/1323, piqa/1332, piqa/134, piqa/1341, piqa/1344, piqa/1346, piqa/1350, piqa/1352, piqa/1366, piqa/1370, piqa/1372, piqa/1375, piqa/138, piqa/1381, piqa/1383, piqa/1394, piqa/1410, piqa/1412, piqa/1413, piqa/1415, piqa/1417, piqa/1421, piqa/1427, piqa/1439, piqa/1440, piqa/1445, piqa/1447, piqa/1448, piqa/146, piqa/1461, piqa/1469, piqa/1477, piqa/1481, piqa/1485, piqa/1493, piqa/1499, piqa/1501, piqa/1508, piqa/1515, piqa/1517, piqa/1521, piqa/1545, piqa/1553, piqa/156, piqa/1561, piqa/1587, piqa/161, piqa/1613, piqa/1615, piqa/1623, piqa/1638, piqa/1641, piqa/1647, piqa/1648, piqa/1654, piqa/1656, piqa/1660, piqa/1663, piqa/1683, piqa/1689, piqa/1702, piqa/1714, piqa/1728, piqa/173, piqa/1751, piqa/1766, piqa/1772, piqa/1775, piqa/1786, piqa/179, piqa/1790, piqa/1804, piqa/1815, piqa/1836, piqa/184, piqa/186, piqa/188, piqa/189, piqa/197, piqa/198, piqa/206, piqa/209, piqa/22, piqa/223, piqa/23, piqa/230, piqa/234, piqa/243, piqa/244, piqa/256, piqa/257, piqa/271, piqa/272, piqa/284, piqa/288, piqa/308, piqa/313, piqa/319, piqa/325, piqa/328, piqa/329, piqa/33, piqa/332, piqa/35, piqa/366, piqa/369, piqa/377, piqa/388, piqa/397, piqa/402, piqa/409, piqa/413, piqa/423, piqa/427, piqa/435, piqa/443, piqa/455, piqa/47, piqa/471, piqa/485, piqa/490, piqa/492, piqa/493, piqa/497, piqa/498, piqa/500, piqa/505, piqa/515, piqa/524, piqa/54, piqa/546, piqa/553, piqa/554, piqa/564, piqa/566, piqa/570, piqa/572, piqa/573, piqa/576, piqa/579, piqa/589, piqa/593, piqa/601, piqa/613, piqa/616, piqa/617, piqa/620, piqa/626, piqa/630, piqa/631, piqa/638, piqa/64, piqa/643, piqa/646, piqa/652, piqa/655, piqa/656, piqa/666, piqa/67, piqa/680, piqa/681, piqa/688, piqa/689, piqa/698, piqa/706, piqa/708, piqa/71, piqa/711, piqa/714, piqa/719, piqa/730, piqa/747, piqa/756, piqa/758, piqa/762, piqa/770, piqa/774, piqa/778, piqa/79, piqa/796, piqa/798, piqa/809, piqa/814, piqa/816, piqa/817, piqa/823, piqa/848, piqa/85, piqa/850, piqa/855, piqa/857, piqa/876, piqa/888, piqa/894, piqa/899, piqa/930, piqa/939, piqa/946, piqa/949, piqa/957, piqa/971, piqa/973, piqa/984, piqa/993, piqa/997

Problems solved by 1 model only

example_link model min_pass1_of_model
piqa/803 seed30841 0.776
piqa/926 seed30841 0.776
piqa/968 seed30841 0.776
piqa/980 seed30841 0.776
piqa/1538 seed30841 0.776
piqa/46 seed30841 0.776
piqa/1793 seed16385 0.773
piqa/606 seed16385 0.773
piqa/1464 seed16385 0.773
piqa/1527 seed16385 0.773
piqa/1763 seed16385 0.773
piqa/1351 seed16385 0.773
piqa/1070 seed17495 0.773
piqa/873 seed17495 0.773
piqa/453 seed17495 0.773
piqa/1498 seed17495 0.773
piqa/1335 seed149 0.770
piqa/352 seed149 0.770
piqa/686 seed14309 0.769
piqa/336 seed14309 0.769
piqa/842 seed14309 0.769
piqa/14 seed14309 0.769
piqa/1659 seed6529 0.769
piqa/517 seed6529 0.769
piqa/1048 seed24762 0.768
piqa/753 seed24762 0.768
piqa/95 seed24762 0.768
piqa/1176 seed24762 0.768
piqa/41 seed24762 0.768
piqa/611 seed24734 0.768
piqa/1242 seed24734 0.768
piqa/1476 seed24734 0.768
piqa/1132 seed6115 0.766
piqa/124 seed6115 0.766
piqa/1167 seed27712 0.761
piqa/292 seed27712 0.761
piqa/502 seed27712 0.761

Suspect problems

These are 10 problems with the lowest correlation with the overall evaluation (i.e. better models tend to do worse on these. )

example_link pass1_of_ex tau
piqa/1642 0.800 -0.596
piqa/1313 0.700 -0.553
piqa/1463 0.800 -0.522
piqa/1247 0.800 -0.522
piqa/291 0.200 -0.522
piqa/1559 0.200 -0.522
piqa/1042 0.800 -0.522
piqa/1630 0.500 -0.507
piqa/582 0.400 -0.487
piqa/1348 0.900 -0.447

Histogram of accuracies

Histogram of problems by the accuracy on each problem.

Histogram of difficulties

Histogram of problems by the minimum win rate to solve each problem.