There are 885 examples not solved by any model.
Solving some of these can be a good signal that your model is indeed better than leading models if these are good problems.
agi_english/0, agi_english/100, agi_english/1000, agi_english/1002, agi_english/1004, agi_english/1005, agi_english/1006, agi_english/1007, agi_english/1010, agi_english/1012, agi_english/1013, agi_english/1016, agi_english/1018, agi_english/1019, agi_english/102, agi_english/1020, agi_english/1021, agi_english/1022, agi_english/1024, agi_english/1025, agi_english/1027, agi_english/1028, agi_english/103, agi_english/1030, agi_english/1031, agi_english/1033, agi_english/1034, agi_english/1039, agi_english/104, agi_english/1042, agi_english/1043, agi_english/1044, agi_english/1049, agi_english/1054, agi_english/1056, agi_english/1057, agi_english/1060, agi_english/1061, agi_english/1062, agi_english/1064, agi_english/1065, agi_english/1066, agi_english/1067, agi_english/1068, agi_english/1069, agi_english/1071, agi_english/1072, agi_english/1073, agi_english/1074, agi_english/1076, agi_english/1078, agi_english/1080, agi_english/1081, agi_english/1083, agi_english/1085, agi_english/1086, agi_english/1088, agi_english/1089, agi_english/1090, agi_english/1093, agi_english/1094, agi_english/1097, agi_english/1099, agi_english/11, agi_english/110, agi_english/1100, agi_english/1101, agi_english/1102, agi_english/1105, agi_english/1106, agi_english/1107, agi_english/1108, agi_english/1110, agi_english/1112, agi_english/1113, agi_english/1114, agi_english/1117, agi_english/1120, agi_english/1122, agi_english/1123, agi_english/1126, agi_english/1127, agi_english/1128, agi_english/1129, agi_english/1130, agi_english/1134, agi_english/1135, agi_english/1139, agi_english/114, agi_english/1143, agi_english/1145, agi_english/1146, agi_english/1148, agi_english/1150, agi_english/1152, agi_english/1153, agi_english/1154, agi_english/1155, agi_english/1159, agi_english/1160, agi_english/1171, agi_english/1173, agi_english/1174, agi_english/1176, agi_english/1177, agi_english/1178, agi_english/1179, agi_english/118, agi_english/1180, agi_english/1183, agi_english/1189, agi_english/119, agi_english/1193, agi_english/1195, agi_english/1198, agi_english/1200, agi_english/1202, agi_english/1204, agi_english/1207, agi_english/1219, agi_english/1221, agi_english/1224, agi_english/1225, agi_english/1226, agi_english/123, agi_english/1232, agi_english/1233, agi_english/1236, agi_english/124, agi_english/1240, agi_english/1242, agi_english/1245, agi_english/1246, agi_english/1247, agi_english/125, agi_english/1253, agi_english/1254, agi_english/1257, agi_english/1258, agi_english/1260, agi_english/1263, agi_english/1268, agi_english/1269, agi_english/1270, agi_english/1272, agi_english/1274, agi_english/1282, agi_english/1284, agi_english/1285, agi_english/1286, agi_english/1287, agi_english/1288, agi_english/1291, agi_english/1293, agi_english/1296, agi_english/1297, agi_english/1298, agi_english/1299, agi_english/130, agi_english/1303, agi_english/1305, agi_english/1312, agi_english/1315, agi_english/1321, agi_english/1323, agi_english/1324, agi_english/1326, agi_english/1327, agi_english/1329, agi_english/133, agi_english/1335, agi_english/1337, agi_english/1340, agi_english/1341, agi_english/1342, agi_english/1343, agi_english/1345, agi_english/1351, agi_english/1352, agi_english/1355, agi_english/1357, agi_english/1359, agi_english/1360, agi_english/1362, agi_english/1363, agi_english/1367, agi_english/1368, agi_english/137, agi_english/1371, agi_english/1372, agi_english/1373, agi_english/1377, agi_english/1378, agi_english/1379, agi_english/1382, agi_english/1383, agi_english/1386, agi_english/1390, agi_english/1394, agi_english/1396, agi_english/1397, agi_english/1399, agi_english/140, agi_english/1401, agi_english/1402, agi_english/1404, agi_english/1405, agi_english/1407, agi_english/1409, agi_english/1412, agi_english/1414, agi_english/1415, agi_english/1420, agi_english/1422, agi_english/1423, agi_english/1427, agi_english/143, agi_english/1432, agi_english/1434, agi_english/1435, agi_english/1436, agi_english/1440, agi_english/1447, agi_english/1448, agi_english/1449, agi_english/145, agi_english/1450, agi_english/1452, agi_english/1453, agi_english/1455, agi_english/1457, agi_english/1459, agi_english/1460, agi_english/1462, agi_english/1468, agi_english/147, agi_english/1476, agi_english/1477, agi_english/148, agi_english/1481, agi_english/1482, agi_english/1483, agi_english/1485, agi_english/1492, agi_english/1493, agi_english/1499, agi_english/15, agi_english/1501, agi_english/1502, agi_english/1503, agi_english/1505, agi_english/1511, agi_english/1512, agi_english/1513, agi_english/1514, agi_english/1517, agi_english/1523, agi_english/1525, agi_english/1527, agi_english/1530, agi_english/1532, agi_english/1533, agi_english/1535, agi_english/1539, agi_english/1540, agi_english/1541, agi_english/1543, agi_english/1546, agi_english/1549, agi_english/1553, agi_english/1557, agi_english/1559, agi_english/1560, agi_english/1561, agi_english/1563, agi_english/1565, agi_english/1566, agi_english/157, agi_english/1572, agi_english/1573, agi_english/1574, agi_english/1576, agi_english/1578, agi_english/1579, agi_english/1580, agi_english/1584, agi_english/1586, agi_english/1588, agi_english/1589, agi_english/1591, agi_english/1592, agi_english/1597, agi_english/1598, agi_english/1599, agi_english/16, agi_english/1602, agi_english/1609, agi_english/1610, agi_english/1614, agi_english/1616, agi_english/162, agi_english/1624, agi_english/1626, agi_english/1627, agi_english/1630, agi_english/1631, agi_english/1632, agi_english/1633, agi_english/1636, agi_english/1640, agi_english/1642, agi_english/1649, agi_english/1651, agi_english/1653, agi_english/1655, agi_english/1663, agi_english/167, agi_english/1672, agi_english/1676, agi_english/1679, agi_english/168, agi_english/1686, agi_english/169, agi_english/1690, agi_english/1693, agi_english/1696, agi_english/1699, agi_english/170, agi_english/1701, agi_english/1704, agi_english/1707, agi_english/171, agi_english/1711, agi_english/1718, agi_english/172, agi_english/1728, agi_english/1731, agi_english/1733, agi_english/1738, agi_english/1740, agi_english/1741, agi_english/1742, agi_english/1743, agi_english/1745, agi_english/1746, agi_english/1748, agi_english/1750, agi_english/1751, agi_english/1752, agi_english/1753, agi_english/1754, agi_english/1757, agi_english/1758, agi_english/1759, agi_english/1763, agi_english/1765, agi_english/1766, agi_english/1767, agi_english/1768, agi_english/1769, agi_english/1770, agi_english/1772, agi_english/1776, agi_english/1781, agi_english/1785, agi_english/1798, agi_english/1800, agi_english/1803, agi_english/1807, agi_english/181, agi_english/1821, agi_english/1834, agi_english/1838, agi_english/184, agi_english/1840, agi_english/1841, agi_english/1843, agi_english/1846, agi_english/185, agi_english/1850, agi_english/1856, agi_english/1857, agi_english/1858, agi_english/1865, agi_english/1866, agi_english/187, agi_english/1871, agi_english/1876, agi_english/1877, agi_english/1881, agi_english/1886, agi_english/1889, agi_english/1899, agi_english/190, agi_english/1901, agi_english/1904, agi_english/1906, agi_english/1913, agi_english/1914, agi_english/1915, agi_english/1917, agi_english/1918, agi_english/1926, agi_english/1937, agi_english/1943, agi_english/1945, agi_english/195, agi_english/1958, agi_english/1964, agi_english/1966, agi_english/1971, agi_english/1975, agi_english/1985, agi_english/1986, agi_english/1988, agi_english/199, agi_english/1998, agi_english/2, agi_english/2003, agi_english/2008, agi_english/2009, agi_english/2012, agi_english/2017, agi_english/2019, agi_english/202, agi_english/2024, agi_english/2025, agi_english/2040, agi_english/2041, agi_english/2057, agi_english/2087, agi_english/2088, agi_english/2091, agi_english/21, agi_english/210, agi_english/211, agi_english/212, agi_english/2122, agi_english/2124, agi_english/2130, agi_english/2133, agi_english/2136, agi_english/2137, agi_english/214, agi_english/2140, agi_english/2145, agi_english/2146, agi_english/215, agi_english/2150, agi_english/2153, agi_english/2154, agi_english/2155, agi_english/2158, agi_english/2159, agi_english/2163, agi_english/2168, agi_english/2169, agi_english/217, agi_english/2170, agi_english/2171, agi_english/2173, agi_english/2179, agi_english/218, agi_english/2183, agi_english/2185, agi_english/2186, agi_english/2190, agi_english/2191, agi_english/2192, agi_english/2193, agi_english/22, agi_english/2200, agi_english/2201, agi_english/2202, agi_english/2203, agi_english/2204, agi_english/2206, agi_english/2209, agi_english/2211, agi_english/2212, agi_english/2216, agi_english/2221, agi_english/2222, agi_english/2224, agi_english/2228, agi_english/223, agi_english/2232, agi_english/2233, agi_english/2237, agi_english/2238, agi_english/2240, agi_english/2244, agi_english/2245, agi_english/2252, agi_english/2255, agi_english/2256, agi_english/2257, agi_english/226, agi_english/2261, agi_english/2264, agi_english/2267, agi_english/2268, agi_english/2269, agi_english/227, agi_english/2271, agi_english/2273, agi_english/2277, agi_english/2278, agi_english/228, agi_english/2281, agi_english/2282, agi_english/2285, agi_english/2288, agi_english/229, agi_english/2298, agi_english/23, agi_english/2302, agi_english/2303, agi_english/2305, agi_english/2307, agi_english/2308, agi_english/2309, agi_english/2312, agi_english/2313, agi_english/2318, agi_english/232, agi_english/2329, agi_english/233, agi_english/2332, agi_english/2334, agi_english/2338, agi_english/2339, agi_english/2348, agi_english/2351, agi_english/2354, agi_english/2356, agi_english/2359, agi_english/2362, agi_english/2366, agi_english/2372, agi_english/2377, agi_english/2383, agi_english/2390, agi_english/2393, agi_english/2395, agi_english/2398, agi_english/2401, agi_english/2402, agi_english/2404, agi_english/2412, agi_english/2418, agi_english/2419, agi_english/242, agi_english/2425, agi_english/2426, agi_english/243, agi_english/2431, agi_english/2434, agi_english/2437, agi_english/2441, agi_english/2443, agi_english/245, agi_english/2452, agi_english/2456, agi_english/2458, agi_english/2459, agi_english/246, agi_english/2462, agi_english/2468, agi_english/2469, agi_english/248, agi_english/2488, agi_english/2491, agi_english/2492, agi_english/2493, agi_english/2494, agi_english/2496, agi_english/2501, agi_english/2505, agi_english/2508, agi_english/251, agi_english/2514, agi_english/2515, agi_english/2517, agi_english/2522, agi_english/2525, agi_english/2529, agi_english/253, agi_english/2532, agi_english/2534, agi_english/2537, agi_english/2539, agi_english/2540, agi_english/257, agi_english/258, agi_english/260, agi_english/261, agi_english/263, agi_english/265, agi_english/269, agi_english/27, agi_english/273, agi_english/277, agi_english/280, agi_english/281, agi_english/283, agi_english/284, agi_english/287, agi_english/288, agi_english/29, agi_english/291, agi_english/294, agi_english/298, agi_english/300, agi_english/301, agi_english/306, agi_english/307, agi_english/31, agi_english/311, agi_english/312, agi_english/315, agi_english/318, agi_english/321, agi_english/324, agi_english/326, agi_english/327, agi_english/328, agi_english/334, agi_english/336, agi_english/342, agi_english/344, agi_english/347, agi_english/348, agi_english/349, agi_english/35, agi_english/353, agi_english/357, agi_english/36, agi_english/364, agi_english/365, agi_english/369, agi_english/37, agi_english/370, agi_english/375, agi_english/38, agi_english/380, agi_english/383, agi_english/384, agi_english/385, agi_english/387, agi_english/39, agi_english/40, agi_english/400, agi_english/405, agi_english/406, agi_english/41, agi_english/410, agi_english/411, agi_english/422, agi_english/424, agi_english/426, agi_english/428, agi_english/43, agi_english/431, agi_english/433, agi_english/435, agi_english/436, agi_english/438, agi_english/441, agi_english/445, agi_english/447, agi_english/449, agi_english/451, agi_english/455, agi_english/457, agi_english/46, agi_english/461, agi_english/463, agi_english/468, agi_english/469, agi_english/47, agi_english/471, agi_english/478, agi_english/483, agi_english/488, agi_english/491, agi_english/493, agi_english/498, agi_english/499, agi_english/501, agi_english/504, agi_english/505, agi_english/513, agi_english/517, agi_english/521, agi_english/524, agi_english/525, agi_english/526, agi_english/529, agi_english/53, agi_english/535, agi_english/536, agi_english/540, agi_english/541, agi_english/55, agi_english/551, agi_english/556, agi_english/558, agi_english/56, agi_english/562, agi_english/569, agi_english/571, agi_english/576, agi_english/577, agi_english/578, agi_english/582, agi_english/583, agi_english/584, agi_english/585, agi_english/586, agi_english/587, agi_english/592, agi_english/593, agi_english/595, agi_english/599, agi_english/6, agi_english/601, agi_english/605, agi_english/607, agi_english/61, agi_english/611, agi_english/616, agi_english/620, agi_english/624, agi_english/634, agi_english/638, agi_english/639, agi_english/641, agi_english/644, agi_english/650, agi_english/651, agi_english/652, agi_english/656, agi_english/662, agi_english/668, agi_english/673, agi_english/678, agi_english/68, agi_english/680, agi_english/681, agi_english/684, agi_english/689, agi_english/695, agi_english/696, agi_english/699, agi_english/706, agi_english/708, agi_english/71, agi_english/713, agi_english/714, agi_english/716, agi_english/718, agi_english/721, agi_english/723, agi_english/732, agi_english/733, agi_english/736, agi_english/74, agi_english/740, agi_english/742, agi_english/743, agi_english/745, agi_english/748, agi_english/751, agi_english/753, agi_english/756, agi_english/757, agi_english/759, agi_english/76, agi_english/761, agi_english/768, agi_english/774, agi_english/78, agi_english/780, agi_english/786, agi_english/792, agi_english/798, agi_english/799, agi_english/8, agi_english/803, agi_english/804, agi_english/808, agi_english/81, agi_english/815, agi_english/817, agi_english/819, agi_english/82, agi_english/822, agi_english/825, agi_english/828, agi_english/829, agi_english/83, agi_english/832, agi_english/839, agi_english/84, agi_english/846, agi_english/849, agi_english/85, agi_english/850, agi_english/853, agi_english/86, agi_english/860, agi_english/865, agi_english/867, agi_english/871, agi_english/878, agi_english/880, agi_english/885, agi_english/886, agi_english/895, agi_english/897, agi_english/898, agi_english/9, agi_english/901, agi_english/903, agi_english/904, agi_english/905, agi_english/906, agi_english/907, agi_english/909, agi_english/910, agi_english/911, agi_english/913, agi_english/914, agi_english/92, agi_english/920, agi_english/921, agi_english/922, agi_english/924, agi_english/925, agi_english/926, agi_english/928, agi_english/929, agi_english/932, agi_english/936, agi_english/937, agi_english/938, agi_english/939, agi_english/94, agi_english/940, agi_english/943, agi_english/944, agi_english/948, agi_english/95, agi_english/950, agi_english/951, agi_english/953, agi_english/954, agi_english/955, agi_english/958, agi_english/962, agi_english/963, agi_english/964, agi_english/966, agi_english/967, agi_english/968, agi_english/969, agi_english/97, agi_english/970, agi_english/973, agi_english/974, agi_english/977, agi_english/979, agi_english/982, agi_english/983, agi_english/984, agi_english/985, agi_english/986, agi_english/990, agi_english/992, agi_english/994, agi_english/995, agi_english/996, agi_english/997, agi_english/999
| example_link | model | min_pass1_of_model |
|---|---|---|
| agi_english/1168 | seed6529 | 0.223 |
| agi_english/1181 | seed6529 | 0.223 |
| agi_english/1211 | seed6529 | 0.223 |
| agi_english/854 | seed6529 | 0.223 |
| agi_english/2297 | seed6529 | 0.223 |
| agi_english/249 | seed6529 | 0.223 |
| agi_english/2262 | seed6529 | 0.223 |
| agi_english/2259 | seed6529 | 0.223 |
| agi_english/2320 | seed6529 | 0.223 |
| agi_english/2311 | seed6529 | 0.223 |
| agi_english/2172 | seed6529 | 0.223 |
| agi_english/2276 | seed6529 | 0.223 |
| agi_english/2286 | seed6529 | 0.223 |
| agi_english/2139 | seed6529 | 0.223 |
| agi_english/2322 | seed6529 | 0.223 |
| agi_english/13 | seed6529 | 0.223 |
| agi_english/144 | seed6529 | 0.223 |
| agi_english/1437 | seed6529 | 0.223 |
| agi_english/1933 | seed6529 | 0.223 |
| agi_english/550 | seed6529 | 0.223 |
| agi_english/564 | seed6529 | 0.223 |
| agi_english/1692 | seed6529 | 0.223 |
| agi_english/1665 | seed6529 | 0.223 |
| agi_english/1594 | seed6529 | 0.223 |
| agi_english/516 | seed6529 | 0.223 |
| agi_english/543 | seed6529 | 0.223 |
| agi_english/1656 | seed6529 | 0.223 |
| agi_english/335 | seed6529 | 0.223 |
| agi_english/340 | seed6529 | 0.223 |
| agi_english/514 | seed6529 | 0.223 |
| agi_english/267 | seed6529 | 0.223 |
| agi_english/2231 | seed6529 | 0.223 |
| agi_english/2205 | seed6529 | 0.223 |
| agi_english/2194 | seed6529 | 0.223 |
| agi_english/2132 | seed6529 | 0.223 |
| agi_english/676 | seed6529 | 0.223 |
| agi_english/724 | seed6529 | 0.223 |
| agi_english/782 | seed6529 | 0.223 |
| agi_english/830 | seed6529 | 0.223 |
| agi_english/767 | seed6529 | 0.223 |
| agi_english/763 | seed6529 | 0.223 |
| agi_english/2039 | seed6529 | 0.223 |
| agi_english/1999 | seed6529 | 0.223 |
| agi_english/113 | seed6529 | 0.223 |
| agi_english/615 | seed6529 | 0.223 |
| agi_english/1381 | seed6529 | 0.223 |
| agi_english/1228 | seed6529 | 0.223 |
| agi_english/467 | seed6529 | 0.223 |
| agi_english/1959 | seed6529 | 0.223 |
| agi_english/1952 | seed6529 | 0.223 |
| agi_english/1895 | seed6529 | 0.223 |
| agi_english/1991 | seed6529 | 0.223 |
| agi_english/18 | seed6529 | 0.223 |
| agi_english/2005 | seed6529 | 0.223 |
| agi_english/176 | seed6529 | 0.223 |
| agi_english/193 | seed6529 | 0.223 |
| agi_english/890 | seed149 | 0.222 |
| agi_english/2397 | seed149 | 0.222 |
| agi_english/889 | seed149 | 0.222 |
| agi_english/2461 | seed149 | 0.222 |
| agi_english/858 | seed149 | 0.222 |
| agi_english/855 | seed149 | 0.222 |
| agi_english/2405 | seed149 | 0.222 |
| agi_english/2380 | seed149 | 0.222 |
| agi_english/139 | seed149 | 0.222 |
| agi_english/589 | seed149 | 0.222 |
| agi_english/591 | seed149 | 0.222 |
| agi_english/2105 | seed149 | 0.222 |
| agi_english/2357 | seed149 | 0.222 |
| agi_english/2353 | seed149 | 0.222 |
| agi_english/2330 | seed149 | 0.222 |
| agi_english/597 | seed149 | 0.222 |
| agi_english/602 | seed149 | 0.222 |
| agi_english/539 | seed149 | 0.222 |
| agi_english/533 | seed149 | 0.222 |
| agi_english/338 | seed149 | 0.222 |
| agi_english/329 | seed149 | 0.222 |
| agi_english/373 | seed149 | 0.222 |
| agi_english/302 | seed149 | 0.222 |
| agi_english/304 | seed149 | 0.222 |
| agi_english/552 | seed149 | 0.222 |
| agi_english/549 | seed149 | 0.222 |
| agi_english/553 | seed149 | 0.222 |
| agi_english/1720 | seed149 | 0.222 |
| agi_english/487 | seed149 | 0.222 |
| agi_english/520 | seed149 | 0.222 |
| agi_english/518 | seed149 | 0.222 |
| agi_english/356 | seed149 | 0.222 |
| agi_english/359 | seed149 | 0.222 |
| agi_english/366 | seed149 | 0.222 |
| agi_english/372 | seed149 | 0.222 |
| agi_english/322 | seed149 | 0.222 |
| agi_english/319 | seed149 | 0.222 |
| agi_english/310 | seed149 | 0.222 |
| agi_english/2513 | seed149 | 0.222 |
| agi_english/293 | seed149 | 0.222 |
| agi_english/262 | seed149 | 0.222 |
| agi_english/270 | seed149 | 0.222 |
| agi_english/285 | seed149 | 0.222 |
| agi_english/1849 | seed149 | 0.222 |
| agi_english/1867 | seed149 | 0.222 |
| agi_english/2100 | seed149 | 0.222 |
| agi_english/2053 | seed149 | 0.222 |
| agi_english/1791 | seed149 | 0.222 |
| agi_english/1790 | seed149 | 0.222 |
| agi_english/1810 | seed149 | 0.222 |
| agi_english/545 | seed149 | 0.222 |
| agi_english/603 | seed149 | 0.222 |
| agi_english/563 | seed149 | 0.222 |
| agi_english/567 | seed149 | 0.222 |
| agi_english/565 | seed149 | 0.222 |
| agi_english/575 | seed149 | 0.222 |
| agi_english/479 | seed149 | 0.222 |
| agi_english/399 | seed149 | 0.222 |
| agi_english/394 | seed149 | 0.222 |
| agi_english/415 | seed149 | 0.222 |
| agi_english/427 | seed149 | 0.222 |
| agi_english/500 | seed149 | 0.222 |
| agi_english/429 | seed149 | 0.222 |
| agi_english/511 | seed149 | 0.222 |
| agi_english/448 | seed149 | 0.222 |
| agi_english/453 | seed149 | 0.222 |
| agi_english/1892 | seed149 | 0.222 |
| agi_english/432 | seed149 | 0.222 |
| agi_english/443 | seed149 | 0.222 |
| agi_english/382 | seed149 | 0.222 |
| agi_english/379 | seed149 | 0.222 |
| agi_english/462 | seed149 | 0.222 |
| agi_english/401 | seed149 | 0.222 |
| agi_english/414 | seed149 | 0.222 |
| agi_english/1951 | seed149 | 0.222 |
| agi_english/777 | seed149 | 0.222 |
| agi_english/812 | seed149 | 0.222 |
| agi_english/810 | seed149 | 0.222 |
| agi_english/800 | seed149 | 0.222 |
| agi_english/877 | seed149 | 0.222 |
| agi_english/789 | seed149 | 0.222 |
| agi_english/458 | seed149 | 0.222 |
| agi_english/456 | seed149 | 0.222 |
| agi_english/507 | seed149 | 0.222 |
| agi_english/870 | seed149 | 0.222 |
| agi_english/863 | seed149 | 0.222 |
| agi_english/864 | seed149 | 0.222 |
| agi_english/797 | seed149 | 0.222 |
| agi_english/783 | seed149 | 0.222 |
| agi_english/840 | seed149 | 0.222 |
| agi_english/852 | seed149 | 0.222 |
| agi_english/672 | seed149 | 0.222 |
| agi_english/648 | seed149 | 0.222 |
| agi_english/813 | seed149 | 0.222 |
| agi_english/818 | seed149 | 0.222 |
| agi_english/872 | seed149 | 0.222 |
| agi_english/621 | seed149 | 0.222 |
| agi_english/679 | seed149 | 0.222 |
| agi_english/636 | seed149 | 0.222 |
| agi_english/707 | seed149 | 0.222 |
| agi_english/704 | seed149 | 0.222 |
| agi_english/700 | seed149 | 0.222 |
| agi_english/698 | seed149 | 0.222 |
| agi_english/687 | seed149 | 0.222 |
| agi_english/655 | seed149 | 0.222 |
| agi_english/726 | seed149 | 0.222 |
| agi_english/747 | seed149 | 0.222 |
| agi_english/1376 | seed17495 | 0.220 |
| agi_english/1387 | seed17495 | 0.220 |
| agi_english/1316 | seed17495 | 0.220 |
| agi_english/1319 | seed17495 | 0.220 |
| agi_english/1239 | seed17495 | 0.220 |
| agi_english/1210 | seed17495 | 0.220 |
| agi_english/779 | seed17495 | 0.220 |
| agi_english/1531 | seed17495 | 0.220 |
| agi_english/1997 | seed17495 | 0.220 |
| agi_english/1570 | seed17495 | 0.220 |
| agi_english/2054 | seed17495 | 0.220 |
| agi_english/1163 | seed17495 | 0.220 |
| agi_english/2346 | seed17495 | 0.220 |
| agi_english/1261 | seed17495 | 0.220 |
| agi_english/1279 | seed17495 | 0.220 |
| agi_english/1295 | seed17495 | 0.220 |
| agi_english/1262 | seed17495 | 0.220 |
| agi_english/1466 | seed17495 | 0.220 |
| agi_english/1500 | seed17495 | 0.220 |
| agi_english/141 | seed17495 | 0.220 |
| agi_english/1473 | seed17495 | 0.220 |
| agi_english/1474 | seed17495 | 0.220 |
| agi_english/1428 | seed17495 | 0.220 |
| agi_english/1464 | seed17495 | 0.220 |
| agi_english/1471 | seed17495 | 0.220 |
| agi_english/150 | seed17495 | 0.220 |
| agi_english/2472 | seed17495 | 0.220 |
| agi_english/2489 | seed17495 | 0.220 |
| agi_english/2394 | seed17495 | 0.220 |
| agi_english/2498 | seed17495 | 0.220 |
| agi_english/1635 | seed17495 | 0.220 |
| agi_english/1639 | seed17495 | 0.220 |
| agi_english/2542 | seed17495 | 0.220 |
| agi_english/2117 | seed17495 | 0.220 |
| agi_english/1634 | seed17495 | 0.220 |
| agi_english/1548 | seed17495 | 0.220 |
| agi_english/1832 | seed17495 | 0.220 |
| agi_english/1786 | seed17495 | 0.220 |
| agi_english/1521 | seed17495 | 0.220 |
| agi_english/2090 | seed17495 | 0.220 |
| agi_english/674 | seed24762 | 0.216 |
| agi_english/663 | seed24762 | 0.216 |
| agi_english/657 | seed24762 | 0.216 |
| agi_english/642 | seed24762 | 0.216 |
| agi_english/760 | seed24762 | 0.216 |
| agi_english/573 | seed24762 | 0.216 |
| agi_english/2101 | seed24762 | 0.216 |
| agi_english/2079 | seed24762 | 0.216 |
| agi_english/613 | seed24762 | 0.216 |
| agi_english/515 | seed24762 | 0.216 |
| agi_english/2028 | seed24762 | 0.216 |
| agi_english/609 | seed24762 | 0.216 |
| agi_english/1509 | seed24762 | 0.216 |
| agi_english/570 | seed24762 | 0.216 |
| agi_english/492 | seed24762 | 0.216 |
| agi_english/784 | seed24762 | 0.216 |
| agi_english/859 | seed24762 | 0.216 |
| agi_english/653 | seed24762 | 0.216 |
| agi_english/1979 | seed24762 | 0.216 |
| agi_english/1941 | seed24762 | 0.216 |
| agi_english/374 | seed24762 | 0.216 |
| agi_english/381 | seed24762 | 0.216 |
| agi_english/392 | seed24762 | 0.216 |
| agi_english/1954 | seed24762 | 0.216 |
| agi_english/1978 | seed24762 | 0.216 |
| agi_english/1976 | seed24762 | 0.216 |
| agi_english/444 | seed24762 | 0.216 |
| agi_english/476 | seed24762 | 0.216 |
| agi_english/835 | seed24762 | 0.216 |
| agi_english/793 | seed24762 | 0.216 |
| agi_english/833 | seed6115 | 0.215 |
| agi_english/848 | seed6115 | 0.215 |
| agi_english/377 | seed6115 | 0.215 |
| agi_english/1969 | seed6115 | 0.215 |
| agi_english/711 | seed6115 | 0.215 |
| agi_english/664 | seed6115 | 0.215 |
| agi_english/2114 | seed6115 | 0.215 |
| agi_english/2175 | seed6115 | 0.215 |
| agi_english/2110 | seed6115 | 0.215 |
| agi_english/354 | seed6115 | 0.215 |
| agi_english/1508 | seed6115 | 0.215 |
| agi_english/1187 | seed6115 | 0.215 |
| agi_english/1860 | seed6115 | 0.215 |
| agi_english/1864 | seed6115 | 0.215 |
| agi_english/781 | seed6115 | 0.215 |
| agi_english/794 | seed6115 | 0.215 |
| agi_english/1465 | seed6115 | 0.215 |
| agi_english/1488 | seed6115 | 0.215 |
| agi_english/120 | seed6115 | 0.215 |
| agi_english/1 | seed6115 | 0.215 |
| agi_english/1217 | seed6115 | 0.215 |
| agi_english/1199 | seed6115 | 0.215 |
| agi_english/1646 | seed6115 | 0.215 |
| agi_english/2078 | seed6115 | 0.215 |
| agi_english/1680 | seed6115 | 0.215 |
| agi_english/1660 | seed6115 | 0.215 |
| agi_english/1819 | seed6115 | 0.215 |
| agi_english/1617 | seed6115 | 0.215 |
| agi_english/1637 | seed6115 | 0.215 |
| agi_english/1241 | seed14309 | 0.213 |
| agi_english/1870 | seed14309 | 0.213 |
| agi_english/1842 | seed14309 | 0.213 |
| agi_english/1902 | seed14309 | 0.213 |
| agi_english/1925 | seed14309 | 0.213 |
| agi_english/1244 | seed14309 | 0.213 |
| agi_english/1670 | seed14309 | 0.213 |
| agi_english/1761 | seed14309 | 0.213 |
| agi_english/1661 | seed14309 | 0.213 |
| agi_english/2161 | seed14309 | 0.213 |
| agi_english/2147 | seed14309 | 0.213 |
| agi_english/2151 | seed14309 | 0.213 |
| agi_english/2160 | seed14309 | 0.213 |
| agi_english/2195 | seed14309 | 0.213 |
| agi_english/2180 | seed14309 | 0.213 |
| agi_english/2181 | seed14309 | 0.213 |
| agi_english/2182 | seed14309 | 0.213 |
| agi_english/2215 | seed14309 | 0.213 |
| agi_english/2219 | seed14309 | 0.213 |
| agi_english/2536 | seed14309 | 0.213 |
| agi_english/2225 | seed14309 | 0.213 |
| agi_english/2000 | seed14309 | 0.213 |
| agi_english/1654 | seed14309 | 0.213 |
| agi_english/2526 | seed14309 | 0.213 |
| agi_english/1684 | seed14309 | 0.213 |
| agi_english/1813 | seed14309 | 0.213 |
| agi_english/1547 | seed14309 | 0.213 |
| agi_english/1529 | seed14309 | 0.213 |
| agi_english/1989 | seed14309 | 0.213 |
| agi_english/2038 | seed14309 | 0.213 |
| agi_english/2036 | seed14309 | 0.213 |
| agi_english/1771 | seed14309 | 0.213 |
| agi_english/1822 | seed14309 | 0.213 |
| agi_english/1747 | seed14309 | 0.213 |
| agi_english/1735 | seed14309 | 0.213 |
| agi_english/1703 | seed14309 | 0.213 |
| agi_english/1755 | seed14309 | 0.213 |
| agi_english/1310 | seed14309 | 0.213 |
| agi_english/2310 | seed14309 | 0.213 |
| agi_english/1338 | seed14309 | 0.213 |
| agi_english/1251 | seed14309 | 0.213 |
| agi_english/1400 | seed14309 | 0.213 |
| agi_english/1431 | seed14309 | 0.213 |
| agi_english/1278 | seed14309 | 0.213 |
| agi_english/2368 | seed14309 | 0.213 |
| agi_english/2289 | seed14309 | 0.213 |
| agi_english/2131 | seed14309 | 0.213 |
| agi_english/2134 | seed14309 | 0.213 |
| agi_english/2335 | seed14309 | 0.213 |
| agi_english/2325 | seed14309 | 0.213 |
| agi_english/2315 | seed14309 | 0.213 |
| agi_english/2265 | seed14309 | 0.213 |
| agi_english/1426 | seed14309 | 0.213 |
| agi_english/2234 | seed14309 | 0.213 |
| agi_english/2290 | seed14309 | 0.213 |
| agi_english/2292 | seed14309 | 0.213 |
| agi_english/2295 | seed14309 | 0.213 |
| agi_english/2470 | seed14309 | 0.213 |
| agi_english/2466 | seed14309 | 0.213 |
| agi_english/2499 | seed14309 | 0.213 |
| agi_english/2483 | seed14309 | 0.213 |
| agi_english/2484 | seed14309 | 0.213 |
| agi_english/2239 | seed14309 | 0.213 |
| agi_english/2242 | seed14309 | 0.213 |
| agi_english/2474 | seed14309 | 0.213 |
| agi_english/2406 | seed14309 | 0.213 |
| agi_english/2416 | seed14309 | 0.213 |
| agi_english/2440 | seed14309 | 0.213 |
| agi_english/2388 | seed14309 | 0.213 |
| agi_english/1206 | seed14309 | 0.213 |
| agi_english/1208 | seed14309 | 0.213 |
| agi_english/1166 | seed14309 | 0.213 |
| agi_english/959 | seed14309 | 0.213 |
| agi_english/1138 | seed30841 | 0.210 |
| agi_english/1164 | seed30841 | 0.210 |
| agi_english/1162 | seed30841 | 0.210 |
| agi_english/1213 | seed30841 | 0.210 |
| agi_english/1185 | seed30841 | 0.210 |
| agi_english/122 | seed30841 | 0.210 |
| agi_english/1235 | seed30841 | 0.210 |
| agi_english/1250 | seed30841 | 0.210 |
| agi_english/1987 | seed30841 | 0.210 |
| agi_english/2164 | seed30841 | 0.210 |
| agi_english/2199 | seed30841 | 0.210 |
| agi_english/1370 | seed30841 | 0.210 |
| agi_english/1393 | seed30841 | 0.210 |
| agi_english/1536 | seed30841 | 0.210 |
| agi_english/1613 | seed30841 | 0.210 |
| agi_english/1612 | seed30841 | 0.210 |
| agi_english/2026 | seed30841 | 0.210 |
| agi_english/1934 | seed30841 | 0.210 |
| agi_english/2032 | seed30841 | 0.210 |
| agi_english/189 | seed30841 | 0.210 |
| agi_english/1927 | seed30841 | 0.210 |
| agi_english/2001 | seed30841 | 0.210 |
| agi_english/2006 | seed30841 | 0.210 |
| agi_english/2015 | seed30841 | 0.210 |
| agi_english/2174 | seed30841 | 0.210 |
| agi_english/2184 | seed30841 | 0.210 |
| agi_english/1229 | seed27712 | 0.207 |
| agi_english/772 | seed27712 | 0.207 |
| agi_english/1157 | seed27712 | 0.207 |
| agi_english/99 | seed27712 | 0.207 |
| agi_english/106 | seed27712 | 0.207 |
| agi_english/108 | seed27712 | 0.207 |
| agi_english/738 | seed27712 | 0.207 |
| agi_english/670 | seed27712 | 0.207 |
| agi_english/111 | seed27712 | 0.207 |
| agi_english/1137 | seed27712 | 0.207 |
| agi_english/129 | seed27712 | 0.207 |
| agi_english/1375 | seed27712 | 0.207 |
| agi_english/647 | seed27712 | 0.207 |
| agi_english/1325 | seed27712 | 0.207 |
| agi_english/1289 | seed27712 | 0.207 |
| agi_english/1516 | seed27712 | 0.207 |
| agi_english/239 | seed27712 | 0.207 |
| agi_english/806 | seed27712 | 0.207 |
| agi_english/79 | seed27712 | 0.207 |
| agi_english/1212 | seed27712 | 0.207 |
| agi_english/1218 | seed27712 | 0.207 |
| agi_english/1946 | seed27712 | 0.207 |
| agi_english/1929 | seed27712 | 0.207 |
| agi_english/2045 | seed27712 | 0.207 |
| agi_english/2103 | seed27712 | 0.207 |
| agi_english/2094 | seed27712 | 0.207 |
| agi_english/188 | seed27712 | 0.207 |
| agi_english/208 | seed27712 | 0.207 |
| agi_english/2044 | seed27712 | 0.207 |
| agi_english/2047 | seed27712 | 0.207 |
| agi_english/1961 | seed27712 | 0.207 |
| agi_english/186 | seed27712 | 0.207 |
| agi_english/219 | seed27712 | 0.207 |
| agi_english/230 | seed27712 | 0.207 |
| agi_english/2167 | seed27712 | 0.207 |
| agi_english/48 | seed27712 | 0.207 |
| agi_english/1773 | seed27712 | 0.207 |
| agi_english/1789 | seed27712 | 0.207 |
| agi_english/180 | seed27712 | 0.207 |
| agi_english/17 | seed27712 | 0.207 |
| agi_english/5 | seed27712 | 0.207 |
| agi_english/174 | seed27712 | 0.207 |
| agi_english/127 | seed27712 | 0.207 |
| agi_english/135 | seed27712 | 0.207 |
| agi_english/1336 | seed27712 | 0.207 |
| agi_english/1350 | seed27712 | 0.207 |
| agi_english/1332 | seed27712 | 0.207 |
| agi_english/64 | seed27712 | 0.207 |
| agi_english/1439 | seed27712 | 0.207 |
| agi_english/60 | seed27712 | 0.207 |
| agi_english/149 | seed27712 | 0.207 |
| agi_english/153 | seed27712 | 0.207 |
| agi_english/54 | seed27712 | 0.207 |
| agi_english/579 | seed27712 | 0.207 |
| agi_english/57 | seed27712 | 0.207 |
| agi_english/58 | seed27712 | 0.207 |
| agi_english/1603 | seed27712 | 0.207 |
| agi_english/160 | seed27712 | 0.207 |
| agi_english/163 | seed27712 | 0.207 |
| agi_english/156 | seed27712 | 0.207 |
| agi_english/1544 | seed27712 | 0.207 |
| agi_english/1555 | seed27712 | 0.207 |
| agi_english/250 | seed27712 | 0.207 |
| agi_english/25 | seed27712 | 0.207 |
| agi_english/2258 | seed27712 | 0.207 |
| agi_english/235 | seed27712 | 0.207 |
| agi_english/1520 | seed27712 | 0.207 |
| agi_english/1306 | seed27712 | 0.207 |
| agi_english/1256 | seed27712 | 0.207 |
| agi_english/151 | seed27712 | 0.207 |
| agi_english/155 | seed27712 | 0.207 |
| agi_english/2475 | seed24734 | 0.207 |
| agi_english/2524 | seed24734 | 0.207 |
| agi_english/77 | seed24734 | 0.207 |
| agi_english/1793 | seed24734 | 0.207 |
| agi_english/240 | seed24734 | 0.207 |
| agi_english/2415 | seed24734 | 0.207 |
| agi_english/2409 | seed24734 | 0.207 |
| agi_english/2407 | seed24734 | 0.207 |
| agi_english/2429 | seed24734 | 0.207 |
| agi_english/2464 | seed24734 | 0.207 |
| agi_english/2444 | seed24734 | 0.207 |
| agi_english/2438 | seed24734 | 0.207 |
| agi_english/200 | seed24734 | 0.207 |
| agi_english/28 | seed24734 | 0.207 |
| agi_english/2349 | seed24734 | 0.207 |
| agi_english/2486 | seed24734 | 0.207 |
| agi_english/2248 | seed24734 | 0.207 |
| agi_english/221 | seed24734 | 0.207 |
| agi_english/2226 | seed24734 | 0.207 |
| agi_english/2500 | seed24734 | 0.207 |
| agi_english/2482 | seed24734 | 0.207 |
| agi_english/2361 | seed24734 | 0.207 |
| agi_english/2512 | seed24734 | 0.207 |
| agi_english/2509 | seed24734 | 0.207 |
| agi_english/3 | seed24734 | 0.207 |
| agi_english/30 | seed24734 | 0.207 |
| agi_english/2010 | seed24734 | 0.207 |
| agi_english/45 | seed24734 | 0.207 |
| agi_english/142 | seed24734 | 0.207 |
| agi_english/89 | seed24734 | 0.207 |
| agi_english/67 | seed24734 | 0.207 |
| agi_english/128 | seed24734 | 0.207 |
| agi_english/205 | seed24734 | 0.207 |
| agi_english/1862 | seed24734 | 0.207 |
| agi_english/191 | seed24734 | 0.207 |
| agi_english/1828 | seed24734 | 0.207 |
| agi_english/1144 | seed16385 | 0.205 |
| agi_english/1192 | seed16385 | 0.205 |
| agi_english/1215 | seed16385 | 0.205 |
| agi_english/1223 | seed16385 | 0.205 |
| agi_english/1835 | seed16385 | 0.205 |
| agi_english/1921 | seed16385 | 0.205 |
| agi_english/2327 | seed16385 | 0.205 |
| agi_english/1564 | seed16385 | 0.205 |
| agi_english/1490 | seed16385 | 0.205 |
| agi_english/1486 | seed16385 | 0.205 |
| agi_english/1472 | seed16385 | 0.205 |
| agi_english/1458 | seed16385 | 0.205 |
| agi_english/1443 | seed16385 | 0.205 |
| agi_english/1417 | seed16385 | 0.205 |
| agi_english/1398 | seed16385 | 0.205 |
| agi_english/1641 | seed16385 | 0.205 |
| agi_english/1568 | seed16385 | 0.205 |
| agi_english/1596 | seed16385 | 0.205 |
| agi_english/1595 | seed16385 | 0.205 |
| agi_english/1252 | seed16385 | 0.205 |
| agi_english/1313 | seed16385 | 0.205 |
| agi_english/1275 | seed16385 | 0.205 |
| agi_english/1369 | seed16385 | 0.205 |
| agi_english/1380 | seed16385 | 0.205 |
These are 10 problems with the lowest correlation with the overall evaluation (i.e. better models tend to do worse on these. )
| example_link | pass1_of_ex | tau |
|---|---|---|
| agi_english/614 | 0.500 | -0.745 |
| agi_english/339 | 0.400 | -0.730 |
| agi_english/477 | 0.700 | -0.618 |
| agi_english/1883 | 0.300 | -0.618 |
| agi_english/1650 | 0.300 | -0.618 |
| agi_english/308 | 0.700 | -0.618 |
| agi_english/317 | 0.700 | -0.618 |
| agi_english/1201 | 0.300 | -0.618 |
| agi_english/581 | 0.700 | -0.618 |
| agi_english/1880 | 0.300 | -0.618 |
Histogram of problems by the accuracy on each problem.
Histogram of problems by the minimum win rate to solve each problem.