There are 811 examples not solved by any model.
Solving some of these can be a good signal that your model is indeed better than leading models if these are good problems.
siqa/10, siqa/1002, siqa/1005, siqa/1007, siqa/1008, siqa/1009, siqa/1012, siqa/1014, siqa/1018, siqa/102, siqa/1020, siqa/1027, siqa/1031, siqa/1036, siqa/1037, siqa/1038, siqa/104, siqa/1041, siqa/1042, siqa/1045, siqa/1046, siqa/1047, siqa/1048, siqa/1049, siqa/1055, siqa/1058, siqa/106, siqa/1061, siqa/1062, siqa/1063, siqa/1064, siqa/1067, siqa/1068, siqa/1071, siqa/1072, siqa/1073, siqa/1074, siqa/1076, siqa/1077, siqa/1078, siqa/1080, siqa/1081, siqa/1082, siqa/1083, siqa/1085, siqa/1088, siqa/1090, siqa/1091, siqa/1093, siqa/1097, siqa/11, siqa/1104, siqa/1105, siqa/1111, siqa/1113, siqa/1114, siqa/1116, siqa/1118, siqa/1119, siqa/1120, siqa/1121, siqa/1124, siqa/113, siqa/1130, siqa/1131, siqa/1132, siqa/1133, siqa/1134, siqa/1139, siqa/1140, siqa/1141, siqa/1145, siqa/1146, siqa/1148, siqa/1149, siqa/1150, siqa/1152, siqa/1153, siqa/1154, siqa/1156, siqa/1161, siqa/1163, siqa/1164, siqa/1168, siqa/1170, siqa/1171, siqa/1180, siqa/1181, siqa/1186, siqa/1188, siqa/1191, siqa/1192, siqa/1193, siqa/1194, siqa/1198, siqa/1199, siqa/1201, siqa/1204, siqa/1205, siqa/1211, siqa/1216, siqa/1221, siqa/1222, siqa/1223, siqa/1229, siqa/1231, siqa/1232, siqa/1235, siqa/1238, siqa/1241, siqa/1242, siqa/1246, siqa/1250, siqa/1251, siqa/1252, siqa/1256, siqa/1260, siqa/1263, siqa/1265, siqa/1269, siqa/1271, siqa/1272, siqa/1273, siqa/1275, siqa/1276, siqa/1280, siqa/1282, siqa/1283, siqa/1285, siqa/1286, siqa/1287, siqa/129, siqa/1291, siqa/1294, siqa/1304, siqa/1308, siqa/1314, siqa/1315, siqa/1318, siqa/132, siqa/1320, siqa/1321, siqa/1324, siqa/1325, siqa/1326, siqa/1328, siqa/1334, siqa/1337, siqa/134, siqa/1343, siqa/1348, siqa/1350, siqa/1351, siqa/1353, siqa/1355, siqa/1356, siqa/1357, siqa/1358, siqa/1359, siqa/1360, siqa/1361, siqa/1362, siqa/1363, siqa/1365, siqa/137, siqa/1370, siqa/1371, siqa/1373, siqa/1375, siqa/1376, siqa/1378, siqa/1380, siqa/1381, siqa/1385, siqa/1386, siqa/1388, siqa/1390, siqa/1391, siqa/1392, siqa/1401, siqa/1405, siqa/1406, siqa/1407, siqa/1408, siqa/1412, siqa/1413, siqa/1424, siqa/1426, siqa/1428, siqa/1430, siqa/1431, siqa/1432, siqa/1435, siqa/1437, siqa/1438, siqa/144, siqa/1440, siqa/1441, siqa/1443, siqa/1445, siqa/1446, siqa/1447, siqa/1448, siqa/145, siqa/1454, siqa/1459, siqa/146, siqa/1460, siqa/1461, siqa/1462, siqa/1465, siqa/1466, siqa/1467, siqa/147, siqa/1470, siqa/1471, siqa/1472, siqa/1478, siqa/1479, siqa/1481, siqa/1482, siqa/1486, siqa/1488, siqa/1489, siqa/1490, siqa/1495, siqa/1497, siqa/1498, siqa/1499, siqa/1500, siqa/1505, siqa/1507, siqa/1509, siqa/1511, siqa/1512, siqa/1513, siqa/1517, siqa/1518, siqa/1520, siqa/1521, siqa/1523, siqa/1524, siqa/1526, siqa/1527, siqa/153, siqa/1532, siqa/1533, siqa/1535, siqa/1539, siqa/1540, siqa/1544, siqa/1547, siqa/155, siqa/1550, siqa/1551, siqa/1556, siqa/1562, siqa/1564, siqa/1565, siqa/1566, siqa/1567, siqa/1568, siqa/1571, siqa/1580, siqa/1584, siqa/1587, siqa/1589, siqa/1590, siqa/1591, siqa/1593, siqa/1594, siqa/1595, siqa/1599, siqa/1605, siqa/1612, siqa/1615, siqa/1617, siqa/1619, siqa/162, siqa/1625, siqa/1628, siqa/163, siqa/1630, siqa/1632, siqa/1634, siqa/1640, siqa/1641, siqa/1646, siqa/1649, siqa/1650, siqa/1651, siqa/1652, siqa/1654, siqa/1655, siqa/1658, siqa/166, siqa/1663, siqa/167, siqa/1674, siqa/1676, siqa/1685, siqa/1686, siqa/1689, siqa/1692, siqa/1694, siqa/1695, siqa/1697, siqa/1698, siqa/1701, siqa/1702, siqa/1703, siqa/1705, siqa/1709, siqa/1710, siqa/1714, siqa/1716, siqa/1717, siqa/1719, siqa/172, siqa/1724, siqa/1725, siqa/1726, siqa/1729, siqa/1730, siqa/1733, siqa/1735, siqa/1736, siqa/1739, siqa/174, siqa/1740, siqa/1741, siqa/1743, siqa/1744, siqa/1745, siqa/1749, siqa/175, siqa/1750, siqa/1751, siqa/1754, siqa/1759, siqa/1761, siqa/1762, siqa/1766, siqa/1767, siqa/1768, siqa/1769, siqa/177, siqa/1776, siqa/1777, siqa/1778, siqa/1780, siqa/1782, siqa/1784, siqa/1786, siqa/1788, siqa/1790, siqa/1792, siqa/1793, siqa/1794, siqa/1800, siqa/1801, siqa/1803, siqa/1804, siqa/1807, siqa/1808, siqa/1810, siqa/1811, siqa/1812, siqa/1815, siqa/1816, siqa/1817, siqa/1818, siqa/1819, siqa/182, siqa/1820, siqa/1822, siqa/1824, siqa/1825, siqa/1827, siqa/1830, siqa/1832, siqa/1833, siqa/1836, siqa/1837, siqa/1839, siqa/1840, siqa/1843, siqa/1845, siqa/1847, siqa/1849, siqa/1852, siqa/1856, siqa/1859, siqa/1861, siqa/1865, siqa/1866, siqa/1867, siqa/1869, siqa/1879, siqa/1880, siqa/1881, siqa/1882, siqa/1883, siqa/1886, siqa/1891, siqa/1894, siqa/1895, siqa/1896, siqa/1898, siqa/1899, siqa/190, siqa/1900, siqa/1905, siqa/1906, siqa/1909, siqa/1910, siqa/1913, siqa/1919, siqa/1921, siqa/1923, siqa/1926, siqa/1927, siqa/193, siqa/1933, siqa/1934, siqa/1936, siqa/194, siqa/1942, siqa/1944, siqa/1945, siqa/1948, siqa/1949, siqa/195, siqa/1953, siqa/198, siqa/199, siqa/203, siqa/204, siqa/205, siqa/208, siqa/209, siqa/21, siqa/211, siqa/213, siqa/219, siqa/22, siqa/221, siqa/229, siqa/23, siqa/235, siqa/237, siqa/240, siqa/243, siqa/244, siqa/245, siqa/249, siqa/253, siqa/256, siqa/259, siqa/260, siqa/262, siqa/263, siqa/266, siqa/267, siqa/271, siqa/272, siqa/274, siqa/275, siqa/277, siqa/278, siqa/281, siqa/283, siqa/284, siqa/285, siqa/286, siqa/289, siqa/290, siqa/293, siqa/296, siqa/298, siqa/300, siqa/302, siqa/303, siqa/307, siqa/308, siqa/315, siqa/318, siqa/32, siqa/324, siqa/327, siqa/33, siqa/330, siqa/332, siqa/334, siqa/335, siqa/337, siqa/339, siqa/34, siqa/342, siqa/344, siqa/345, siqa/347, siqa/349, siqa/352, siqa/353, siqa/354, siqa/358, siqa/359, siqa/367, siqa/372, siqa/373, siqa/374, siqa/376, siqa/379, siqa/38, siqa/382, siqa/384, siqa/387, siqa/388, siqa/392, siqa/397, siqa/40, siqa/401, siqa/402, siqa/404, siqa/405, siqa/407, siqa/409, siqa/41, siqa/410, siqa/412, siqa/413, siqa/416, siqa/418, siqa/421, siqa/423, siqa/424, siqa/43, siqa/437, siqa/438, siqa/439, siqa/44, siqa/441, siqa/443, siqa/444, siqa/448, siqa/457, siqa/458, siqa/460, siqa/461, siqa/462, siqa/463, siqa/465, siqa/469, siqa/474, siqa/475, siqa/476, siqa/479, siqa/480, siqa/483, siqa/484, siqa/485, siqa/487, siqa/488, siqa/489, siqa/491, siqa/492, siqa/496, siqa/497, siqa/499, siqa/5, siqa/501, siqa/503, siqa/504, siqa/505, siqa/506, siqa/508, siqa/509, siqa/511, siqa/514, siqa/516, siqa/517, siqa/518, siqa/519, siqa/52, siqa/523, siqa/525, siqa/526, siqa/531, siqa/532, siqa/536, siqa/539, siqa/54, siqa/547, siqa/548, siqa/549, siqa/550, siqa/551, siqa/552, siqa/565, siqa/566, siqa/567, siqa/568, siqa/57, siqa/573, siqa/575, siqa/576, siqa/577, siqa/578, siqa/579, siqa/58, siqa/583, siqa/585, siqa/587, siqa/588, siqa/589, siqa/593, siqa/595, siqa/596, siqa/597, siqa/598, siqa/6, siqa/60, siqa/603, siqa/604, siqa/605, siqa/606, siqa/609, siqa/613, siqa/614, siqa/617, siqa/619, siqa/62, siqa/625, siqa/627, siqa/629, siqa/63, siqa/631, siqa/635, siqa/636, siqa/637, siqa/646, siqa/653, siqa/656, siqa/658, siqa/66, siqa/665, siqa/667, siqa/670, siqa/674, siqa/675, siqa/678, siqa/684, siqa/685, siqa/691, siqa/693, siqa/696, siqa/699, siqa/70, siqa/700, siqa/702, siqa/703, siqa/705, siqa/709, siqa/71, siqa/710, siqa/711, siqa/712, siqa/713, siqa/715, siqa/716, siqa/72, siqa/725, siqa/726, siqa/728, siqa/729, siqa/734, siqa/738, siqa/742, siqa/743, siqa/744, siqa/745, siqa/746, siqa/747, siqa/748, siqa/75, siqa/750, siqa/751, siqa/752, siqa/756, siqa/757, siqa/758, siqa/76, siqa/760, siqa/762, siqa/764, siqa/765, siqa/767, siqa/77, siqa/770, siqa/773, siqa/776, siqa/777, siqa/778, siqa/779, siqa/783, siqa/785, siqa/788, siqa/789, siqa/791, siqa/793, siqa/799, siqa/80, siqa/803, siqa/805, siqa/806, siqa/807, siqa/81, siqa/810, siqa/812, siqa/813, siqa/817, siqa/818, siqa/82, siqa/820, siqa/826, siqa/830, siqa/834, siqa/836, siqa/837, siqa/838, siqa/839, siqa/84, siqa/841, siqa/842, siqa/845, siqa/846, siqa/848, siqa/85, siqa/852, siqa/853, siqa/859, siqa/86, siqa/862, siqa/863, siqa/864, siqa/865, siqa/866, siqa/867, siqa/868, siqa/869, siqa/87, siqa/870, siqa/872, siqa/875, siqa/879, siqa/881, siqa/882, siqa/883, siqa/884, siqa/885, siqa/89, siqa/890, siqa/895, siqa/896, siqa/898, siqa/90, siqa/904, siqa/906, siqa/908, siqa/915, siqa/916, siqa/917, siqa/92, siqa/920, siqa/922, siqa/926, siqa/927, siqa/928, siqa/935, siqa/940, siqa/942, siqa/945, siqa/946, siqa/948, siqa/949, siqa/950, siqa/954, siqa/958, siqa/959, siqa/960, siqa/962, siqa/964, siqa/965, siqa/969, siqa/971, siqa/972, siqa/974, siqa/977, siqa/98, siqa/981, siqa/982, siqa/984, siqa/985, siqa/989, siqa/99, siqa/992, siqa/994, siqa/997, siqa/998, siqa/999
| example_link | model | min_pass1_of_model |
|---|---|---|
| siqa/1003 | seed6115 | 0.477 |
| siqa/1379 | seed6115 | 0.477 |
| siqa/931 | seed6115 | 0.477 |
| siqa/1604 | seed6115 | 0.477 |
| siqa/1572 | seed6115 | 0.477 |
| siqa/1202 | seed6115 | 0.477 |
| siqa/1296 | seed6115 | 0.477 |
| siqa/1060 | seed16385 | 0.471 |
| siqa/1764 | seed16385 | 0.471 |
| siqa/477 | seed16385 | 0.471 |
| siqa/481 | seed16385 | 0.471 |
| siqa/680 | seed16385 | 0.471 |
| siqa/707 | seed16385 | 0.471 |
| siqa/1255 | seed16385 | 0.471 |
| siqa/173 | seed16385 | 0.471 |
| siqa/1573 | seed16385 | 0.471 |
| siqa/1339 | seed30841 | 0.470 |
| siqa/255 | seed30841 | 0.470 |
| siqa/1862 | seed30841 | 0.470 |
| siqa/1422 | seed30841 | 0.470 |
| siqa/1352 | seed30841 | 0.470 |
| siqa/1366 | seed30841 | 0.470 |
| siqa/1553 | seed30841 | 0.470 |
| siqa/1492 | seed30841 | 0.470 |
| siqa/1331 | seed30841 | 0.470 |
| siqa/78 | seed14309 | 0.468 |
| siqa/943 | seed17495 | 0.468 |
| siqa/821 | seed14309 | 0.468 |
| siqa/1028 | seed14309 | 0.468 |
| siqa/1100 | seed14309 | 0.468 |
| siqa/873 | seed24762 | 0.468 |
| siqa/1720 | seed24762 | 0.468 |
| siqa/1887 | seed24762 | 0.468 |
| siqa/1237 | seed17495 | 0.468 |
| siqa/1578 | seed17495 | 0.468 |
| siqa/1757 | seed17495 | 0.468 |
| siqa/135 | seed24762 | 0.468 |
| siqa/542 | seed24762 | 0.468 |
| siqa/623 | seed24762 | 0.468 |
| siqa/39 | seed24762 | 0.468 |
| siqa/197 | seed14309 | 0.468 |
| siqa/390 | seed14309 | 0.468 |
| siqa/1939 | seed14309 | 0.468 |
| siqa/1870 | seed17495 | 0.468 |
| siqa/888 | seed14309 | 0.468 |
| siqa/486 | seed24762 | 0.468 |
| siqa/218 | seed14309 | 0.468 |
| siqa/1016 | seed6529 | 0.464 |
| siqa/1143 | seed149 | 0.464 |
| siqa/1054 | seed6529 | 0.464 |
| siqa/1010 | seed149 | 0.464 |
| siqa/1585 | seed6529 | 0.464 |
| siqa/158 | seed6529 | 0.464 |
| siqa/1549 | seed149 | 0.464 |
| siqa/1665 | seed149 | 0.464 |
| siqa/1732 | seed6529 | 0.464 |
| siqa/1851 | seed6529 | 0.464 |
| siqa/735 | seed6529 | 0.464 |
| siqa/257 | seed149 | 0.464 |
| siqa/408 | seed6529 | 0.464 |
| siqa/903 | seed6529 | 0.464 |
| siqa/1848 | seed149 | 0.464 |
| siqa/35 | seed149 | 0.464 |
| siqa/1434 | seed24734 | 0.463 |
| siqa/149 | seed24734 | 0.463 |
| siqa/18 | seed24734 | 0.463 |
| siqa/1752 | seed24734 | 0.463 |
| siqa/1711 | seed24734 | 0.463 |
| siqa/386 | seed24734 | 0.463 |
| siqa/822 | seed24734 | 0.463 |
| siqa/1261 | seed27712 | 0.457 |
| siqa/1374 | seed27712 | 0.457 |
These are 10 problems with the lowest correlation with the overall evaluation (i.e. better models tend to do worse on these. )
| example_link | pass1_of_ex | tau |
|---|---|---|
| siqa/1712 | 0.300 | -0.682 |
| siqa/1455 | 0.800 | -0.625 |
| siqa/905 | 0.800 | -0.625 |
| siqa/1598 | 0.400 | -0.606 |
| siqa/893 | 0.800 | -0.547 |
| siqa/894 | 0.600 | -0.542 |
| siqa/615 | 0.300 | -0.511 |
| siqa/1354 | 0.700 | -0.511 |
| siqa/527 | 0.300 | -0.511 |
| siqa/507 | 0.200 | -0.508 |
Histogram of problems by the accuracy on each problem.
Histogram of problems by the minimum win rate to solve each problem.