siqa_step02500: by examples

Results Paper Code


Not solved by any model

There are 903 examples not solved by any model. Solving some of these can be a good signal that your model is indeed better than leading models if these are good problems.
siqa/0, siqa/1, siqa/10, siqa/1002, siqa/1007, siqa/1008, siqa/1012, siqa/1014, siqa/1018, siqa/102, siqa/1020, siqa/1026, siqa/1028, siqa/103, siqa/1031, siqa/1033, siqa/1036, siqa/1037, siqa/1038, siqa/104, siqa/1040, siqa/1041, siqa/1042, siqa/1046, siqa/1047, siqa/1049, siqa/1050, siqa/1051, siqa/1055, siqa/1058, siqa/106, siqa/1060, siqa/1061, siqa/1062, siqa/1063, siqa/1064, siqa/1067, siqa/1068, siqa/1071, siqa/1072, siqa/1073, siqa/1074, siqa/1076, siqa/1077, siqa/1078, siqa/1079, siqa/108, siqa/1080, siqa/1081, siqa/1082, siqa/1083, siqa/1085, siqa/1088, siqa/1090, siqa/1091, siqa/1092, siqa/1093, siqa/1097, siqa/11, siqa/1100, siqa/1104, siqa/1105, siqa/111, siqa/1111, siqa/1113, siqa/1114, siqa/1116, siqa/1118, siqa/1119, siqa/1120, siqa/1121, siqa/1124, siqa/1127, siqa/1128, siqa/113, siqa/1130, siqa/1131, siqa/1132, siqa/1133, siqa/1134, siqa/1139, siqa/1140, siqa/1141, siqa/1145, siqa/1146, siqa/1148, siqa/1149, siqa/1150, siqa/1152, siqa/1153, siqa/1154, siqa/1156, siqa/1161, siqa/1163, siqa/1164, siqa/1168, siqa/1169, siqa/1172, siqa/1181, siqa/1186, siqa/1191, siqa/1193, siqa/1194, siqa/1198, siqa/1199, siqa/1200, siqa/1201, siqa/1202, siqa/1204, siqa/1205, siqa/1210, siqa/1212, siqa/1214, siqa/1216, siqa/1221, siqa/1223, siqa/1226, siqa/1229, siqa/1230, siqa/1231, siqa/1232, siqa/1235, siqa/1237, siqa/1238, siqa/1241, siqa/1242, siqa/125, siqa/1250, siqa/1251, siqa/1252, siqa/1256, siqa/1260, siqa/1263, siqa/1265, siqa/1268, siqa/1269, siqa/1271, siqa/1272, siqa/1273, siqa/1274, siqa/1275, siqa/1276, siqa/1280, siqa/1282, siqa/1283, siqa/1285, siqa/1286, siqa/129, siqa/1291, siqa/1293, siqa/1294, siqa/1296, siqa/1306, siqa/1308, siqa/1309, siqa/1314, siqa/1315, siqa/1318, siqa/132, siqa/1321, siqa/1324, siqa/1325, siqa/1326, siqa/1328, siqa/1331, siqa/1334, siqa/1337, siqa/134, siqa/1344, siqa/1348, siqa/1350, siqa/1351, siqa/1352, siqa/1353, siqa/1355, siqa/1356, siqa/1357, siqa/1358, siqa/1359, siqa/1360, siqa/1361, siqa/1362, siqa/1363, siqa/1364, siqa/1365, siqa/1366, siqa/1369, siqa/137, siqa/1370, siqa/1371, siqa/1372, siqa/1373, siqa/1374, siqa/1375, siqa/1376, siqa/1378, siqa/1379, siqa/1380, siqa/1381, siqa/1382, siqa/1385, siqa/1386, siqa/1388, siqa/1390, siqa/1391, siqa/1392, siqa/1397, siqa/14, siqa/1401, siqa/1405, siqa/1406, siqa/1408, siqa/1412, siqa/1413, siqa/1414, siqa/142, siqa/1423, siqa/1424, siqa/1425, siqa/1426, siqa/1427, siqa/1428, siqa/1430, siqa/1431, siqa/1432, siqa/1434, siqa/1435, siqa/1437, siqa/144, siqa/1440, siqa/1441, siqa/1442, siqa/1443, siqa/1445, siqa/1447, siqa/1448, siqa/145, siqa/1455, siqa/1457, siqa/1459, siqa/146, siqa/1460, siqa/1461, siqa/1462, siqa/1465, siqa/1466, siqa/1467, siqa/147, siqa/1470, siqa/1471, siqa/1472, siqa/1478, siqa/1479, siqa/1481, siqa/1482, siqa/1483, siqa/1485, siqa/1486, siqa/1488, siqa/1489, siqa/149, siqa/1490, siqa/1492, siqa/1495, siqa/1497, siqa/1498, siqa/1499, siqa/1500, siqa/1505, siqa/1507, siqa/1509, siqa/1511, siqa/1512, siqa/1513, siqa/1514, siqa/1517, siqa/1518, siqa/1520, siqa/1521, siqa/1523, siqa/1524, siqa/1526, siqa/1527, siqa/153, siqa/1530, siqa/1532, siqa/1533, siqa/1535, siqa/1537, siqa/1539, siqa/1540, siqa/1542, siqa/1544, siqa/1546, siqa/1547, siqa/1548, siqa/1550, siqa/1551, siqa/1553, siqa/1555, siqa/1556, siqa/1562, siqa/1564, siqa/1565, siqa/1566, siqa/1567, siqa/1568, siqa/1571, siqa/1572, siqa/1573, siqa/1580, siqa/1585, siqa/1587, siqa/1589, siqa/1591, siqa/1593, siqa/1594, siqa/1595, siqa/1604, siqa/1605, siqa/1608, siqa/161, siqa/1612, siqa/1615, siqa/1617, siqa/1619, siqa/162, siqa/1626, siqa/1628, siqa/163, siqa/1630, siqa/1632, siqa/1633, siqa/1634, siqa/1638, siqa/1640, siqa/1641, siqa/1646, siqa/1647, siqa/1649, siqa/1650, siqa/1651, siqa/1652, siqa/1654, siqa/1655, siqa/1657, siqa/1658, siqa/166, siqa/1663, siqa/167, siqa/1674, siqa/1676, siqa/1689, siqa/1692, siqa/1694, siqa/1695, siqa/1697, siqa/1698, siqa/1701, siqa/1702, siqa/1703, siqa/1705, siqa/1709, siqa/1710, siqa/1712, siqa/1714, siqa/1719, siqa/172, siqa/1720, siqa/1724, siqa/1725, siqa/1726, siqa/1729, siqa/173, siqa/1730, siqa/1732, siqa/1733, siqa/1735, siqa/1737, siqa/1739, siqa/174, siqa/1740, siqa/1741, siqa/1743, siqa/1744, siqa/1745, siqa/1748, siqa/1749, siqa/175, siqa/1750, siqa/1751, siqa/1752, siqa/1754, siqa/1757, siqa/1759, siqa/1760, siqa/1761, siqa/1762, siqa/1763, siqa/1764, siqa/1766, siqa/1767, siqa/1768, siqa/1769, siqa/177, siqa/1776, siqa/1777, siqa/1778, siqa/178, siqa/1780, siqa/1782, siqa/1784, siqa/1786, siqa/1790, siqa/1792, siqa/1794, siqa/1796, siqa/18, siqa/1800, siqa/1801, siqa/1803, siqa/1804, siqa/1805, siqa/1807, siqa/1808, siqa/181, siqa/1810, siqa/1811, siqa/1815, siqa/1816, siqa/1817, siqa/1818, siqa/182, siqa/1820, siqa/1822, siqa/1824, siqa/1825, siqa/1827, siqa/1830, siqa/1833, siqa/1837, siqa/1839, siqa/1843, siqa/1845, siqa/1847, siqa/1848, siqa/1851, siqa/1852, siqa/1855, siqa/1856, siqa/1859, siqa/1861, siqa/1862, siqa/1865, siqa/1866, siqa/1867, siqa/1872, siqa/1879, siqa/1880, siqa/1882, siqa/1883, siqa/1886, siqa/1891, siqa/1894, siqa/1895, siqa/1896, siqa/1898, siqa/1899, siqa/190, siqa/1900, siqa/1905, siqa/1906, siqa/1909, siqa/1910, siqa/1913, siqa/1916, siqa/1918, siqa/1919, siqa/192, siqa/1921, siqa/1923, siqa/1926, siqa/1927, siqa/193, siqa/1933, siqa/1934, siqa/1936, siqa/1939, siqa/194, siqa/1942, siqa/1944, siqa/1945, siqa/1949, siqa/195, siqa/1953, siqa/197, siqa/198, siqa/204, siqa/208, siqa/209, siqa/211, siqa/213, siqa/218, siqa/219, siqa/22, siqa/220, siqa/221, siqa/224, siqa/229, siqa/23, siqa/235, siqa/237, siqa/240, siqa/243, siqa/244, siqa/245, siqa/248, siqa/249, siqa/253, siqa/255, siqa/256, siqa/259, siqa/260, siqa/262, siqa/263, siqa/267, siqa/271, siqa/272, siqa/275, siqa/276, siqa/277, siqa/278, siqa/280, siqa/281, siqa/283, siqa/284, siqa/285, siqa/286, siqa/289, siqa/290, siqa/292, siqa/293, siqa/296, siqa/298, siqa/3, siqa/300, siqa/302, siqa/303, siqa/307, siqa/308, siqa/31, siqa/311, siqa/313, siqa/315, siqa/318, siqa/32, siqa/324, siqa/33, siqa/330, siqa/332, siqa/334, siqa/335, siqa/337, siqa/339, siqa/34, siqa/342, siqa/344, siqa/345, siqa/346, siqa/347, siqa/349, siqa/35, siqa/352, siqa/353, siqa/354, siqa/360, siqa/367, siqa/372, siqa/373, siqa/374, siqa/376, siqa/378, siqa/379, siqa/38, siqa/382, siqa/384, siqa/385, siqa/387, siqa/388, siqa/390, siqa/397, siqa/40, siqa/401, siqa/402, siqa/404, siqa/405, siqa/407, siqa/409, siqa/410, siqa/413, siqa/416, siqa/418, siqa/420, siqa/421, siqa/423, siqa/424, siqa/43, siqa/438, siqa/439, siqa/44, siqa/441, siqa/443, siqa/444, siqa/448, siqa/454, siqa/455, siqa/456, siqa/460, siqa/461, siqa/462, siqa/463, siqa/465, siqa/469, siqa/470, siqa/472, siqa/474, siqa/475, siqa/476, siqa/477, siqa/479, siqa/480, siqa/481, siqa/483, siqa/484, siqa/485, siqa/486, siqa/487, siqa/488, siqa/489, siqa/491, siqa/492, siqa/496, siqa/497, siqa/5, siqa/501, siqa/503, siqa/504, siqa/505, siqa/506, siqa/508, siqa/51, siqa/511, siqa/514, siqa/516, siqa/518, siqa/519, siqa/52, siqa/523, siqa/524, siqa/525, siqa/526, siqa/531, siqa/532, siqa/533, siqa/536, siqa/539, siqa/54, siqa/545, siqa/547, siqa/549, siqa/550, siqa/551, siqa/558, siqa/559, siqa/565, siqa/566, siqa/567, siqa/568, siqa/57, siqa/571, siqa/573, siqa/575, siqa/576, siqa/577, siqa/578, siqa/579, siqa/58, siqa/580, siqa/583, siqa/587, siqa/588, siqa/589, siqa/591, siqa/593, siqa/595, siqa/596, siqa/597, siqa/598, siqa/599, siqa/6, siqa/60, siqa/603, siqa/604, siqa/605, siqa/606, siqa/608, siqa/609, siqa/613, siqa/619, siqa/62, siqa/621, siqa/625, siqa/626, siqa/627, siqa/629, siqa/63, siqa/631, siqa/635, siqa/636, siqa/637, siqa/646, siqa/655, siqa/656, siqa/658, siqa/66, siqa/665, siqa/667, siqa/670, siqa/674, siqa/675, siqa/678, siqa/680, siqa/684, siqa/685, siqa/691, siqa/692, siqa/693, siqa/699, siqa/70, siqa/702, siqa/703, siqa/704, siqa/705, siqa/709, siqa/71, siqa/710, siqa/711, siqa/712, siqa/713, siqa/715, siqa/716, siqa/72, siqa/722, siqa/725, siqa/726, siqa/728, siqa/729, siqa/734, siqa/735, siqa/738, siqa/739, siqa/742, siqa/743, siqa/744, siqa/746, siqa/747, siqa/748, siqa/750, siqa/751, siqa/752, siqa/756, siqa/758, siqa/76, siqa/765, siqa/767, siqa/77, siqa/770, siqa/773, siqa/777, siqa/778, siqa/783, siqa/785, siqa/788, siqa/789, siqa/790, siqa/791, siqa/793, siqa/799, siqa/8, siqa/80, siqa/802, siqa/803, siqa/805, siqa/806, siqa/807, siqa/81, siqa/810, siqa/812, siqa/813, siqa/815, siqa/817, siqa/818, siqa/82, siqa/822, siqa/825, siqa/826, siqa/828, siqa/830, siqa/834, siqa/836, siqa/837, siqa/838, siqa/839, siqa/84, siqa/841, siqa/842, siqa/845, siqa/846, siqa/848, siqa/85, siqa/851, siqa/852, siqa/853, siqa/855, siqa/86, siqa/862, siqa/863, siqa/864, siqa/865, siqa/866, siqa/867, siqa/868, siqa/869, siqa/87, siqa/870, siqa/872, siqa/873, siqa/875, siqa/88, siqa/880, siqa/881, siqa/882, siqa/883, siqa/884, siqa/885, siqa/89, siqa/890, siqa/894, siqa/896, siqa/898, siqa/9, siqa/90, siqa/901, siqa/902, siqa/903, siqa/904, siqa/906, siqa/908, siqa/910, siqa/912, siqa/915, siqa/916, siqa/917, siqa/92, siqa/920, siqa/922, siqa/925, siqa/926, siqa/927, siqa/928, siqa/935, siqa/94, siqa/940, siqa/942, siqa/943, siqa/945, siqa/946, siqa/947, siqa/948, siqa/950, siqa/952, siqa/954, siqa/955, siqa/958, siqa/959, siqa/960, siqa/961, siqa/962, siqa/965, siqa/969, siqa/971, siqa/972, siqa/977, siqa/981, siqa/982, siqa/985, siqa/989, siqa/99, siqa/992, siqa/994, siqa/997, siqa/998, siqa/999

Problems solved by 1 model only

example_link model min_pass1_of_model
siqa/1056 seed14309 0.432
siqa/1180 seed14309 0.432
siqa/232 seed14309 0.432
siqa/412 seed14309 0.432
siqa/1736 seed14309 0.432
siqa/1841 seed14309 0.432
siqa/1915 seed14309 0.432
siqa/203 seed14309 0.432
siqa/1887 seed14309 0.432
siqa/1410 seed14309 0.432
siqa/652 seed14309 0.432
siqa/617 seed14309 0.432
siqa/98 seed14309 0.432
siqa/779 seed14309 0.432
siqa/1715 seed24762 0.428
siqa/78 seed24762 0.428
siqa/1143 seed24762 0.428
siqa/1167 seed24762 0.428
siqa/29 seed24762 0.428
siqa/899 seed24762 0.428
siqa/155 seed24762 0.428
siqa/1429 seed24762 0.428
siqa/1059 seed16385 0.424
siqa/1005 seed16385 0.424
siqa/21 seed16385 0.424
siqa/1788 seed16385 0.424
siqa/1948 seed16385 0.424
siqa/1590 seed16385 0.424
siqa/1711 seed16385 0.424
siqa/408 seed16385 0.424
siqa/68 seed16385 0.424
siqa/976 seed16385 0.424
siqa/988 seed16385 0.424
siqa/1246 seed24734 0.423
siqa/662 seed24734 0.423
siqa/907 seed24734 0.423
siqa/964 seed27712 0.423
siqa/1940 seed27712 0.423
siqa/1287 seed27712 0.423
siqa/1377 seed24734 0.423
siqa/1255 seed27712 0.423
siqa/131 seed27712 0.423
siqa/615 seed24734 0.423
siqa/158 seed24734 0.423
siqa/645 seed24734 0.423
siqa/707 seed27712 0.423
siqa/1785 seed17495 0.422
siqa/1295 seed17495 0.422
siqa/1596 seed17495 0.422
siqa/468 seed17495 0.422
siqa/1454 seed17495 0.422
siqa/1438 seed6529 0.421
siqa/264 seed6529 0.421
siqa/41 seed149 0.420
siqa/653 seed149 0.420
siqa/681 seed149 0.420
siqa/1625 seed149 0.420
siqa/1716 seed149 0.420
siqa/1668 seed149 0.420
siqa/1284 seed149 0.420
siqa/517 seed149 0.420
siqa/1881 seed149 0.420
siqa/682 seed30841 0.420
siqa/1160 seed30841 0.420
siqa/1087 seed30841 0.420
siqa/1835 seed30841 0.420
siqa/55 seed30841 0.420
siqa/913 seed30841 0.420
siqa/274 seed30841 0.420
siqa/1176 seed6115 0.414
siqa/1529 seed6115 0.414
siqa/1624 seed6115 0.414
siqa/430 seed6115 0.414
siqa/495 seed6115 0.414

Suspect problems

These are 10 problems with the lowest correlation with the overall evaluation (i.e. better models tend to do worse on these. )

example_link pass1_of_ex tau
siqa/428 0.700 -0.691
siqa/100 0.200 -0.603
siqa/1222 0.200 -0.603
siqa/1795 0.800 -0.603
siqa/939 0.200 -0.603
siqa/970 0.400 -0.585
siqa/118 0.400 -0.585
siqa/1643 0.600 -0.554
siqa/1188 0.500 -0.543
siqa/861 0.800 -0.528

Histogram of accuracies

Histogram of problems by the accuracy on each problem.

Histogram of difficulties

Histogram of problems by the minimum win rate to solve each problem.