agi_english_step02500: by examples

Results Paper Code


Not solved by any model

There are 885 examples not solved by any model. Solving some of these can be a good signal that your model is indeed better than leading models if these are good problems.
agi_english/0, agi_english/100, agi_english/1000, agi_english/1002, agi_english/1004, agi_english/1005, agi_english/1006, agi_english/1007, agi_english/1010, agi_english/1012, agi_english/1013, agi_english/1016, agi_english/1018, agi_english/1019, agi_english/102, agi_english/1020, agi_english/1021, agi_english/1022, agi_english/1024, agi_english/1025, agi_english/1027, agi_english/1028, agi_english/103, agi_english/1030, agi_english/1031, agi_english/1033, agi_english/1034, agi_english/1039, agi_english/104, agi_english/1042, agi_english/1043, agi_english/1044, agi_english/1049, agi_english/1054, agi_english/1056, agi_english/1057, agi_english/1060, agi_english/1061, agi_english/1062, agi_english/1064, agi_english/1065, agi_english/1066, agi_english/1067, agi_english/1068, agi_english/1069, agi_english/1071, agi_english/1072, agi_english/1073, agi_english/1074, agi_english/1076, agi_english/1078, agi_english/1080, agi_english/1081, agi_english/1083, agi_english/1085, agi_english/1086, agi_english/1088, agi_english/1089, agi_english/1090, agi_english/1093, agi_english/1094, agi_english/1097, agi_english/1099, agi_english/11, agi_english/110, agi_english/1100, agi_english/1101, agi_english/1102, agi_english/1105, agi_english/1106, agi_english/1107, agi_english/1108, agi_english/1110, agi_english/1112, agi_english/1113, agi_english/1114, agi_english/1117, agi_english/1120, agi_english/1122, agi_english/1123, agi_english/1126, agi_english/1127, agi_english/1128, agi_english/1129, agi_english/1130, agi_english/1134, agi_english/1135, agi_english/1139, agi_english/114, agi_english/1143, agi_english/1145, agi_english/1146, agi_english/1148, agi_english/1150, agi_english/1152, agi_english/1153, agi_english/1154, agi_english/1155, agi_english/1159, agi_english/1160, agi_english/1171, agi_english/1173, agi_english/1174, agi_english/1176, agi_english/1177, agi_english/1178, agi_english/1179, agi_english/118, agi_english/1180, agi_english/1183, agi_english/1189, agi_english/119, agi_english/1193, agi_english/1195, agi_english/1198, agi_english/1200, agi_english/1202, agi_english/1204, agi_english/1207, agi_english/1219, agi_english/1221, agi_english/1224, agi_english/1225, agi_english/1226, agi_english/123, agi_english/1232, agi_english/1233, agi_english/1236, agi_english/124, agi_english/1240, agi_english/1242, agi_english/1245, agi_english/1246, agi_english/1247, agi_english/125, agi_english/1253, agi_english/1254, agi_english/1257, agi_english/1258, agi_english/1260, agi_english/1263, agi_english/1268, agi_english/1269, agi_english/1270, agi_english/1272, agi_english/1274, agi_english/1282, agi_english/1284, agi_english/1285, agi_english/1286, agi_english/1287, agi_english/1288, agi_english/1291, agi_english/1293, agi_english/1296, agi_english/1297, agi_english/1298, agi_english/1299, agi_english/130, agi_english/1303, agi_english/1305, agi_english/1312, agi_english/1315, agi_english/1321, agi_english/1323, agi_english/1324, agi_english/1326, agi_english/1327, agi_english/1329, agi_english/133, agi_english/1335, agi_english/1337, agi_english/1340, agi_english/1341, agi_english/1342, agi_english/1343, agi_english/1345, agi_english/1351, agi_english/1352, agi_english/1355, agi_english/1357, agi_english/1359, agi_english/1360, agi_english/1362, agi_english/1363, agi_english/1367, agi_english/1368, agi_english/137, agi_english/1371, agi_english/1372, agi_english/1373, agi_english/1377, agi_english/1378, agi_english/1379, agi_english/1382, agi_english/1383, agi_english/1386, agi_english/1390, agi_english/1394, agi_english/1396, agi_english/1397, agi_english/1399, agi_english/140, agi_english/1401, agi_english/1402, agi_english/1404, agi_english/1405, agi_english/1407, agi_english/1409, agi_english/1412, agi_english/1414, agi_english/1415, agi_english/1420, agi_english/1422, agi_english/1423, agi_english/1427, agi_english/143, agi_english/1432, agi_english/1434, agi_english/1435, agi_english/1436, agi_english/1440, agi_english/1447, agi_english/1448, agi_english/1449, agi_english/145, agi_english/1450, agi_english/1452, agi_english/1453, agi_english/1455, agi_english/1457, agi_english/1459, agi_english/1460, agi_english/1462, agi_english/1468, agi_english/147, agi_english/1476, agi_english/1477, agi_english/148, agi_english/1481, agi_english/1482, agi_english/1483, agi_english/1485, agi_english/1492, agi_english/1493, agi_english/1499, agi_english/15, agi_english/1501, agi_english/1502, agi_english/1503, agi_english/1505, agi_english/1511, agi_english/1512, agi_english/1513, agi_english/1514, agi_english/1517, agi_english/1523, agi_english/1525, agi_english/1527, agi_english/1530, agi_english/1532, agi_english/1533, agi_english/1535, agi_english/1539, agi_english/1540, agi_english/1541, agi_english/1543, agi_english/1546, agi_english/1549, agi_english/1553, agi_english/1557, agi_english/1559, agi_english/1560, agi_english/1561, agi_english/1563, agi_english/1565, agi_english/1566, agi_english/157, agi_english/1572, agi_english/1573, agi_english/1574, agi_english/1576, agi_english/1578, agi_english/1579, agi_english/1580, agi_english/1584, agi_english/1586, agi_english/1588, agi_english/1589, agi_english/1591, agi_english/1592, agi_english/1597, agi_english/1598, agi_english/1599, agi_english/16, agi_english/1602, agi_english/1609, agi_english/1610, agi_english/1614, agi_english/1616, agi_english/162, agi_english/1624, agi_english/1626, agi_english/1627, agi_english/1630, agi_english/1631, agi_english/1632, agi_english/1633, agi_english/1636, agi_english/1640, agi_english/1642, agi_english/1649, agi_english/1651, agi_english/1653, agi_english/1655, agi_english/1663, agi_english/167, agi_english/1672, agi_english/1676, agi_english/1679, agi_english/168, agi_english/1686, agi_english/169, agi_english/1690, agi_english/1693, agi_english/1696, agi_english/1699, agi_english/170, agi_english/1701, agi_english/1704, agi_english/1707, agi_english/171, agi_english/1711, agi_english/1718, agi_english/172, agi_english/1728, agi_english/1731, agi_english/1733, agi_english/1738, agi_english/1740, agi_english/1741, agi_english/1742, agi_english/1743, agi_english/1745, agi_english/1746, agi_english/1748, agi_english/1750, agi_english/1751, agi_english/1752, agi_english/1753, agi_english/1754, agi_english/1757, agi_english/1758, agi_english/1759, agi_english/1763, agi_english/1765, agi_english/1766, agi_english/1767, agi_english/1768, agi_english/1769, agi_english/1770, agi_english/1772, agi_english/1776, agi_english/1781, agi_english/1785, agi_english/1798, agi_english/1800, agi_english/1803, agi_english/1807, agi_english/181, agi_english/1821, agi_english/1834, agi_english/1838, agi_english/184, agi_english/1840, agi_english/1841, agi_english/1843, agi_english/1846, agi_english/185, agi_english/1850, agi_english/1856, agi_english/1857, agi_english/1858, agi_english/1865, agi_english/1866, agi_english/187, agi_english/1871, agi_english/1876, agi_english/1877, agi_english/1881, agi_english/1886, agi_english/1889, agi_english/1899, agi_english/190, agi_english/1901, agi_english/1904, agi_english/1906, agi_english/1913, agi_english/1914, agi_english/1915, agi_english/1917, agi_english/1918, agi_english/1926, agi_english/1937, agi_english/1943, agi_english/1945, agi_english/195, agi_english/1958, agi_english/1964, agi_english/1966, agi_english/1971, agi_english/1975, agi_english/1985, agi_english/1986, agi_english/1988, agi_english/199, agi_english/1998, agi_english/2, agi_english/2003, agi_english/2008, agi_english/2009, agi_english/2012, agi_english/2017, agi_english/2019, agi_english/202, agi_english/2024, agi_english/2025, agi_english/2040, agi_english/2041, agi_english/2057, agi_english/2087, agi_english/2088, agi_english/2091, agi_english/21, agi_english/210, agi_english/211, agi_english/212, agi_english/2122, agi_english/2124, agi_english/2130, agi_english/2133, agi_english/2136, agi_english/2137, agi_english/214, agi_english/2140, agi_english/2145, agi_english/2146, agi_english/215, agi_english/2150, agi_english/2153, agi_english/2154, agi_english/2155, agi_english/2158, agi_english/2159, agi_english/2163, agi_english/2168, agi_english/2169, agi_english/217, agi_english/2170, agi_english/2171, agi_english/2173, agi_english/2179, agi_english/218, agi_english/2183, agi_english/2185, agi_english/2186, agi_english/2190, agi_english/2191, agi_english/2192, agi_english/2193, agi_english/22, agi_english/2200, agi_english/2201, agi_english/2202, agi_english/2203, agi_english/2204, agi_english/2206, agi_english/2209, agi_english/2211, agi_english/2212, agi_english/2216, agi_english/2221, agi_english/2222, agi_english/2224, agi_english/2228, agi_english/223, agi_english/2232, agi_english/2233, agi_english/2237, agi_english/2238, agi_english/2240, agi_english/2244, agi_english/2245, agi_english/2252, agi_english/2255, agi_english/2256, agi_english/2257, agi_english/226, agi_english/2261, agi_english/2264, agi_english/2267, agi_english/2268, agi_english/2269, agi_english/227, agi_english/2271, agi_english/2273, agi_english/2277, agi_english/2278, agi_english/228, agi_english/2281, agi_english/2282, agi_english/2285, agi_english/2288, agi_english/229, agi_english/2298, agi_english/23, agi_english/2302, agi_english/2303, agi_english/2305, agi_english/2307, agi_english/2308, agi_english/2309, agi_english/2312, agi_english/2313, agi_english/2318, agi_english/232, agi_english/2329, agi_english/233, agi_english/2332, agi_english/2334, agi_english/2338, agi_english/2339, agi_english/2348, agi_english/2351, agi_english/2354, agi_english/2356, agi_english/2359, agi_english/2362, agi_english/2366, agi_english/2372, agi_english/2377, agi_english/2383, agi_english/2390, agi_english/2393, agi_english/2395, agi_english/2398, agi_english/2401, agi_english/2402, agi_english/2404, agi_english/2412, agi_english/2418, agi_english/2419, agi_english/242, agi_english/2425, agi_english/2426, agi_english/243, agi_english/2431, agi_english/2434, agi_english/2437, agi_english/2441, agi_english/2443, agi_english/245, agi_english/2452, agi_english/2456, agi_english/2458, agi_english/2459, agi_english/246, agi_english/2462, agi_english/2468, agi_english/2469, agi_english/248, agi_english/2488, agi_english/2491, agi_english/2492, agi_english/2493, agi_english/2494, agi_english/2496, agi_english/2501, agi_english/2505, agi_english/2508, agi_english/251, agi_english/2514, agi_english/2515, agi_english/2517, agi_english/2522, agi_english/2525, agi_english/2529, agi_english/253, agi_english/2532, agi_english/2534, agi_english/2537, agi_english/2539, agi_english/2540, agi_english/257, agi_english/258, agi_english/260, agi_english/261, agi_english/263, agi_english/265, agi_english/269, agi_english/27, agi_english/273, agi_english/277, agi_english/280, agi_english/281, agi_english/283, agi_english/284, agi_english/287, agi_english/288, agi_english/29, agi_english/291, agi_english/294, agi_english/298, agi_english/300, agi_english/301, agi_english/306, agi_english/307, agi_english/31, agi_english/311, agi_english/312, agi_english/315, agi_english/318, agi_english/321, agi_english/324, agi_english/326, agi_english/327, agi_english/328, agi_english/334, agi_english/336, agi_english/342, agi_english/344, agi_english/347, agi_english/348, agi_english/349, agi_english/35, agi_english/353, agi_english/357, agi_english/36, agi_english/364, agi_english/365, agi_english/369, agi_english/37, agi_english/370, agi_english/375, agi_english/38, agi_english/380, agi_english/383, agi_english/384, agi_english/385, agi_english/387, agi_english/39, agi_english/40, agi_english/400, agi_english/405, agi_english/406, agi_english/41, agi_english/410, agi_english/411, agi_english/422, agi_english/424, agi_english/426, agi_english/428, agi_english/43, agi_english/431, agi_english/433, agi_english/435, agi_english/436, agi_english/438, agi_english/441, agi_english/445, agi_english/447, agi_english/449, agi_english/451, agi_english/455, agi_english/457, agi_english/46, agi_english/461, agi_english/463, agi_english/468, agi_english/469, agi_english/47, agi_english/471, agi_english/478, agi_english/483, agi_english/488, agi_english/491, agi_english/493, agi_english/498, agi_english/499, agi_english/501, agi_english/504, agi_english/505, agi_english/513, agi_english/517, agi_english/521, agi_english/524, agi_english/525, agi_english/526, agi_english/529, agi_english/53, agi_english/535, agi_english/536, agi_english/540, agi_english/541, agi_english/55, agi_english/551, agi_english/556, agi_english/558, agi_english/56, agi_english/562, agi_english/569, agi_english/571, agi_english/576, agi_english/577, agi_english/578, agi_english/582, agi_english/583, agi_english/584, agi_english/585, agi_english/586, agi_english/587, agi_english/592, agi_english/593, agi_english/595, agi_english/599, agi_english/6, agi_english/601, agi_english/605, agi_english/607, agi_english/61, agi_english/611, agi_english/616, agi_english/620, agi_english/624, agi_english/634, agi_english/638, agi_english/639, agi_english/641, agi_english/644, agi_english/650, agi_english/651, agi_english/652, agi_english/656, agi_english/662, agi_english/668, agi_english/673, agi_english/678, agi_english/68, agi_english/680, agi_english/681, agi_english/684, agi_english/689, agi_english/695, agi_english/696, agi_english/699, agi_english/706, agi_english/708, agi_english/71, agi_english/713, agi_english/714, agi_english/716, agi_english/718, agi_english/721, agi_english/723, agi_english/732, agi_english/733, agi_english/736, agi_english/74, agi_english/740, agi_english/742, agi_english/743, agi_english/745, agi_english/748, agi_english/751, agi_english/753, agi_english/756, agi_english/757, agi_english/759, agi_english/76, agi_english/761, agi_english/768, agi_english/774, agi_english/78, agi_english/780, agi_english/786, agi_english/792, agi_english/798, agi_english/799, agi_english/8, agi_english/803, agi_english/804, agi_english/808, agi_english/81, agi_english/815, agi_english/817, agi_english/819, agi_english/82, agi_english/822, agi_english/825, agi_english/828, agi_english/829, agi_english/83, agi_english/832, agi_english/839, agi_english/84, agi_english/846, agi_english/849, agi_english/85, agi_english/850, agi_english/853, agi_english/86, agi_english/860, agi_english/865, agi_english/867, agi_english/871, agi_english/878, agi_english/880, agi_english/885, agi_english/886, agi_english/895, agi_english/897, agi_english/898, agi_english/9, agi_english/901, agi_english/903, agi_english/904, agi_english/905, agi_english/906, agi_english/907, agi_english/909, agi_english/910, agi_english/911, agi_english/913, agi_english/914, agi_english/92, agi_english/920, agi_english/921, agi_english/922, agi_english/924, agi_english/925, agi_english/926, agi_english/928, agi_english/929, agi_english/932, agi_english/936, agi_english/937, agi_english/938, agi_english/939, agi_english/94, agi_english/940, agi_english/943, agi_english/944, agi_english/948, agi_english/95, agi_english/950, agi_english/951, agi_english/953, agi_english/954, agi_english/955, agi_english/958, agi_english/962, agi_english/963, agi_english/964, agi_english/966, agi_english/967, agi_english/968, agi_english/969, agi_english/97, agi_english/970, agi_english/973, agi_english/974, agi_english/977, agi_english/979, agi_english/982, agi_english/983, agi_english/984, agi_english/985, agi_english/986, agi_english/990, agi_english/992, agi_english/994, agi_english/995, agi_english/996, agi_english/997, agi_english/999

Problems solved by 1 model only

example_link model min_pass1_of_model
agi_english/1168 seed6529 0.223
agi_english/1181 seed6529 0.223
agi_english/1211 seed6529 0.223
agi_english/854 seed6529 0.223
agi_english/2297 seed6529 0.223
agi_english/249 seed6529 0.223
agi_english/2262 seed6529 0.223
agi_english/2259 seed6529 0.223
agi_english/2320 seed6529 0.223
agi_english/2311 seed6529 0.223
agi_english/2172 seed6529 0.223
agi_english/2276 seed6529 0.223
agi_english/2286 seed6529 0.223
agi_english/2139 seed6529 0.223
agi_english/2322 seed6529 0.223
agi_english/13 seed6529 0.223
agi_english/144 seed6529 0.223
agi_english/1437 seed6529 0.223
agi_english/1933 seed6529 0.223
agi_english/550 seed6529 0.223
agi_english/564 seed6529 0.223
agi_english/1692 seed6529 0.223
agi_english/1665 seed6529 0.223
agi_english/1594 seed6529 0.223
agi_english/516 seed6529 0.223
agi_english/543 seed6529 0.223
agi_english/1656 seed6529 0.223
agi_english/335 seed6529 0.223
agi_english/340 seed6529 0.223
agi_english/514 seed6529 0.223
agi_english/267 seed6529 0.223
agi_english/2231 seed6529 0.223
agi_english/2205 seed6529 0.223
agi_english/2194 seed6529 0.223
agi_english/2132 seed6529 0.223
agi_english/676 seed6529 0.223
agi_english/724 seed6529 0.223
agi_english/782 seed6529 0.223
agi_english/830 seed6529 0.223
agi_english/767 seed6529 0.223
agi_english/763 seed6529 0.223
agi_english/2039 seed6529 0.223
agi_english/1999 seed6529 0.223
agi_english/113 seed6529 0.223
agi_english/615 seed6529 0.223
agi_english/1381 seed6529 0.223
agi_english/1228 seed6529 0.223
agi_english/467 seed6529 0.223
agi_english/1959 seed6529 0.223
agi_english/1952 seed6529 0.223
agi_english/1895 seed6529 0.223
agi_english/1991 seed6529 0.223
agi_english/18 seed6529 0.223
agi_english/2005 seed6529 0.223
agi_english/176 seed6529 0.223
agi_english/193 seed6529 0.223
agi_english/890 seed149 0.222
agi_english/2397 seed149 0.222
agi_english/889 seed149 0.222
agi_english/2461 seed149 0.222
agi_english/858 seed149 0.222
agi_english/855 seed149 0.222
agi_english/2405 seed149 0.222
agi_english/2380 seed149 0.222
agi_english/139 seed149 0.222
agi_english/589 seed149 0.222
agi_english/591 seed149 0.222
agi_english/2105 seed149 0.222
agi_english/2357 seed149 0.222
agi_english/2353 seed149 0.222
agi_english/2330 seed149 0.222
agi_english/597 seed149 0.222
agi_english/602 seed149 0.222
agi_english/539 seed149 0.222
agi_english/533 seed149 0.222
agi_english/338 seed149 0.222
agi_english/329 seed149 0.222
agi_english/373 seed149 0.222
agi_english/302 seed149 0.222
agi_english/304 seed149 0.222
agi_english/552 seed149 0.222
agi_english/549 seed149 0.222
agi_english/553 seed149 0.222
agi_english/1720 seed149 0.222
agi_english/487 seed149 0.222
agi_english/520 seed149 0.222
agi_english/518 seed149 0.222
agi_english/356 seed149 0.222
agi_english/359 seed149 0.222
agi_english/366 seed149 0.222
agi_english/372 seed149 0.222
agi_english/322 seed149 0.222
agi_english/319 seed149 0.222
agi_english/310 seed149 0.222
agi_english/2513 seed149 0.222
agi_english/293 seed149 0.222
agi_english/262 seed149 0.222
agi_english/270 seed149 0.222
agi_english/285 seed149 0.222
agi_english/1849 seed149 0.222
agi_english/1867 seed149 0.222
agi_english/2100 seed149 0.222
agi_english/2053 seed149 0.222
agi_english/1791 seed149 0.222
agi_english/1790 seed149 0.222
agi_english/1810 seed149 0.222
agi_english/545 seed149 0.222
agi_english/603 seed149 0.222
agi_english/563 seed149 0.222
agi_english/567 seed149 0.222
agi_english/565 seed149 0.222
agi_english/575 seed149 0.222
agi_english/479 seed149 0.222
agi_english/399 seed149 0.222
agi_english/394 seed149 0.222
agi_english/415 seed149 0.222
agi_english/427 seed149 0.222
agi_english/500 seed149 0.222
agi_english/429 seed149 0.222
agi_english/511 seed149 0.222
agi_english/448 seed149 0.222
agi_english/453 seed149 0.222
agi_english/1892 seed149 0.222
agi_english/432 seed149 0.222
agi_english/443 seed149 0.222
agi_english/382 seed149 0.222
agi_english/379 seed149 0.222
agi_english/462 seed149 0.222
agi_english/401 seed149 0.222
agi_english/414 seed149 0.222
agi_english/1951 seed149 0.222
agi_english/777 seed149 0.222
agi_english/812 seed149 0.222
agi_english/810 seed149 0.222
agi_english/800 seed149 0.222
agi_english/877 seed149 0.222
agi_english/789 seed149 0.222
agi_english/458 seed149 0.222
agi_english/456 seed149 0.222
agi_english/507 seed149 0.222
agi_english/870 seed149 0.222
agi_english/863 seed149 0.222
agi_english/864 seed149 0.222
agi_english/797 seed149 0.222
agi_english/783 seed149 0.222
agi_english/840 seed149 0.222
agi_english/852 seed149 0.222
agi_english/672 seed149 0.222
agi_english/648 seed149 0.222
agi_english/813 seed149 0.222
agi_english/818 seed149 0.222
agi_english/872 seed149 0.222
agi_english/621 seed149 0.222
agi_english/679 seed149 0.222
agi_english/636 seed149 0.222
agi_english/707 seed149 0.222
agi_english/704 seed149 0.222
agi_english/700 seed149 0.222
agi_english/698 seed149 0.222
agi_english/687 seed149 0.222
agi_english/655 seed149 0.222
agi_english/726 seed149 0.222
agi_english/747 seed149 0.222
agi_english/1376 seed17495 0.220
agi_english/1387 seed17495 0.220
agi_english/1316 seed17495 0.220
agi_english/1319 seed17495 0.220
agi_english/1239 seed17495 0.220
agi_english/1210 seed17495 0.220
agi_english/779 seed17495 0.220
agi_english/1531 seed17495 0.220
agi_english/1997 seed17495 0.220
agi_english/1570 seed17495 0.220
agi_english/2054 seed17495 0.220
agi_english/1163 seed17495 0.220
agi_english/2346 seed17495 0.220
agi_english/1261 seed17495 0.220
agi_english/1279 seed17495 0.220
agi_english/1295 seed17495 0.220
agi_english/1262 seed17495 0.220
agi_english/1466 seed17495 0.220
agi_english/1500 seed17495 0.220
agi_english/141 seed17495 0.220
agi_english/1473 seed17495 0.220
agi_english/1474 seed17495 0.220
agi_english/1428 seed17495 0.220
agi_english/1464 seed17495 0.220
agi_english/1471 seed17495 0.220
agi_english/150 seed17495 0.220
agi_english/2472 seed17495 0.220
agi_english/2489 seed17495 0.220
agi_english/2394 seed17495 0.220
agi_english/2498 seed17495 0.220
agi_english/1635 seed17495 0.220
agi_english/1639 seed17495 0.220
agi_english/2542 seed17495 0.220
agi_english/2117 seed17495 0.220
agi_english/1634 seed17495 0.220
agi_english/1548 seed17495 0.220
agi_english/1832 seed17495 0.220
agi_english/1786 seed17495 0.220
agi_english/1521 seed17495 0.220
agi_english/2090 seed17495 0.220
agi_english/674 seed24762 0.216
agi_english/663 seed24762 0.216
agi_english/657 seed24762 0.216
agi_english/642 seed24762 0.216
agi_english/760 seed24762 0.216
agi_english/573 seed24762 0.216
agi_english/2101 seed24762 0.216
agi_english/2079 seed24762 0.216
agi_english/613 seed24762 0.216
agi_english/515 seed24762 0.216
agi_english/2028 seed24762 0.216
agi_english/609 seed24762 0.216
agi_english/1509 seed24762 0.216
agi_english/570 seed24762 0.216
agi_english/492 seed24762 0.216
agi_english/784 seed24762 0.216
agi_english/859 seed24762 0.216
agi_english/653 seed24762 0.216
agi_english/1979 seed24762 0.216
agi_english/1941 seed24762 0.216
agi_english/374 seed24762 0.216
agi_english/381 seed24762 0.216
agi_english/392 seed24762 0.216
agi_english/1954 seed24762 0.216
agi_english/1978 seed24762 0.216
agi_english/1976 seed24762 0.216
agi_english/444 seed24762 0.216
agi_english/476 seed24762 0.216
agi_english/835 seed24762 0.216
agi_english/793 seed24762 0.216
agi_english/833 seed6115 0.215
agi_english/848 seed6115 0.215
agi_english/377 seed6115 0.215
agi_english/1969 seed6115 0.215
agi_english/711 seed6115 0.215
agi_english/664 seed6115 0.215
agi_english/2114 seed6115 0.215
agi_english/2175 seed6115 0.215
agi_english/2110 seed6115 0.215
agi_english/354 seed6115 0.215
agi_english/1508 seed6115 0.215
agi_english/1187 seed6115 0.215
agi_english/1860 seed6115 0.215
agi_english/1864 seed6115 0.215
agi_english/781 seed6115 0.215
agi_english/794 seed6115 0.215
agi_english/1465 seed6115 0.215
agi_english/1488 seed6115 0.215
agi_english/120 seed6115 0.215
agi_english/1 seed6115 0.215
agi_english/1217 seed6115 0.215
agi_english/1199 seed6115 0.215
agi_english/1646 seed6115 0.215
agi_english/2078 seed6115 0.215
agi_english/1680 seed6115 0.215
agi_english/1660 seed6115 0.215
agi_english/1819 seed6115 0.215
agi_english/1617 seed6115 0.215
agi_english/1637 seed6115 0.215
agi_english/1241 seed14309 0.213
agi_english/1870 seed14309 0.213
agi_english/1842 seed14309 0.213
agi_english/1902 seed14309 0.213
agi_english/1925 seed14309 0.213
agi_english/1244 seed14309 0.213
agi_english/1670 seed14309 0.213
agi_english/1761 seed14309 0.213
agi_english/1661 seed14309 0.213
agi_english/2161 seed14309 0.213
agi_english/2147 seed14309 0.213
agi_english/2151 seed14309 0.213
agi_english/2160 seed14309 0.213
agi_english/2195 seed14309 0.213
agi_english/2180 seed14309 0.213
agi_english/2181 seed14309 0.213
agi_english/2182 seed14309 0.213
agi_english/2215 seed14309 0.213
agi_english/2219 seed14309 0.213
agi_english/2536 seed14309 0.213
agi_english/2225 seed14309 0.213
agi_english/2000 seed14309 0.213
agi_english/1654 seed14309 0.213
agi_english/2526 seed14309 0.213
agi_english/1684 seed14309 0.213
agi_english/1813 seed14309 0.213
agi_english/1547 seed14309 0.213
agi_english/1529 seed14309 0.213
agi_english/1989 seed14309 0.213
agi_english/2038 seed14309 0.213
agi_english/2036 seed14309 0.213
agi_english/1771 seed14309 0.213
agi_english/1822 seed14309 0.213
agi_english/1747 seed14309 0.213
agi_english/1735 seed14309 0.213
agi_english/1703 seed14309 0.213
agi_english/1755 seed14309 0.213
agi_english/1310 seed14309 0.213
agi_english/2310 seed14309 0.213
agi_english/1338 seed14309 0.213
agi_english/1251 seed14309 0.213
agi_english/1400 seed14309 0.213
agi_english/1431 seed14309 0.213
agi_english/1278 seed14309 0.213
agi_english/2368 seed14309 0.213
agi_english/2289 seed14309 0.213
agi_english/2131 seed14309 0.213
agi_english/2134 seed14309 0.213
agi_english/2335 seed14309 0.213
agi_english/2325 seed14309 0.213
agi_english/2315 seed14309 0.213
agi_english/2265 seed14309 0.213
agi_english/1426 seed14309 0.213
agi_english/2234 seed14309 0.213
agi_english/2290 seed14309 0.213
agi_english/2292 seed14309 0.213
agi_english/2295 seed14309 0.213
agi_english/2470 seed14309 0.213
agi_english/2466 seed14309 0.213
agi_english/2499 seed14309 0.213
agi_english/2483 seed14309 0.213
agi_english/2484 seed14309 0.213
agi_english/2239 seed14309 0.213
agi_english/2242 seed14309 0.213
agi_english/2474 seed14309 0.213
agi_english/2406 seed14309 0.213
agi_english/2416 seed14309 0.213
agi_english/2440 seed14309 0.213
agi_english/2388 seed14309 0.213
agi_english/1206 seed14309 0.213
agi_english/1208 seed14309 0.213
agi_english/1166 seed14309 0.213
agi_english/959 seed14309 0.213
agi_english/1138 seed30841 0.210
agi_english/1164 seed30841 0.210
agi_english/1162 seed30841 0.210
agi_english/1213 seed30841 0.210
agi_english/1185 seed30841 0.210
agi_english/122 seed30841 0.210
agi_english/1235 seed30841 0.210
agi_english/1250 seed30841 0.210
agi_english/1987 seed30841 0.210
agi_english/2164 seed30841 0.210
agi_english/2199 seed30841 0.210
agi_english/1370 seed30841 0.210
agi_english/1393 seed30841 0.210
agi_english/1536 seed30841 0.210
agi_english/1613 seed30841 0.210
agi_english/1612 seed30841 0.210
agi_english/2026 seed30841 0.210
agi_english/1934 seed30841 0.210
agi_english/2032 seed30841 0.210
agi_english/189 seed30841 0.210
agi_english/1927 seed30841 0.210
agi_english/2001 seed30841 0.210
agi_english/2006 seed30841 0.210
agi_english/2015 seed30841 0.210
agi_english/2174 seed30841 0.210
agi_english/2184 seed30841 0.210
agi_english/1229 seed27712 0.207
agi_english/772 seed27712 0.207
agi_english/1157 seed27712 0.207
agi_english/99 seed27712 0.207
agi_english/106 seed27712 0.207
agi_english/108 seed27712 0.207
agi_english/738 seed27712 0.207
agi_english/670 seed27712 0.207
agi_english/111 seed27712 0.207
agi_english/1137 seed27712 0.207
agi_english/129 seed27712 0.207
agi_english/1375 seed27712 0.207
agi_english/647 seed27712 0.207
agi_english/1325 seed27712 0.207
agi_english/1289 seed27712 0.207
agi_english/1516 seed27712 0.207
agi_english/239 seed27712 0.207
agi_english/806 seed27712 0.207
agi_english/79 seed27712 0.207
agi_english/1212 seed27712 0.207
agi_english/1218 seed27712 0.207
agi_english/1946 seed27712 0.207
agi_english/1929 seed27712 0.207
agi_english/2045 seed27712 0.207
agi_english/2103 seed27712 0.207
agi_english/2094 seed27712 0.207
agi_english/188 seed27712 0.207
agi_english/208 seed27712 0.207
agi_english/2044 seed27712 0.207
agi_english/2047 seed27712 0.207
agi_english/1961 seed27712 0.207
agi_english/186 seed27712 0.207
agi_english/219 seed27712 0.207
agi_english/230 seed27712 0.207
agi_english/2167 seed27712 0.207
agi_english/48 seed27712 0.207
agi_english/1773 seed27712 0.207
agi_english/1789 seed27712 0.207
agi_english/180 seed27712 0.207
agi_english/17 seed27712 0.207
agi_english/5 seed27712 0.207
agi_english/174 seed27712 0.207
agi_english/127 seed27712 0.207
agi_english/135 seed27712 0.207
agi_english/1336 seed27712 0.207
agi_english/1350 seed27712 0.207
agi_english/1332 seed27712 0.207
agi_english/64 seed27712 0.207
agi_english/1439 seed27712 0.207
agi_english/60 seed27712 0.207
agi_english/149 seed27712 0.207
agi_english/153 seed27712 0.207
agi_english/54 seed27712 0.207
agi_english/579 seed27712 0.207
agi_english/57 seed27712 0.207
agi_english/58 seed27712 0.207
agi_english/1603 seed27712 0.207
agi_english/160 seed27712 0.207
agi_english/163 seed27712 0.207
agi_english/156 seed27712 0.207
agi_english/1544 seed27712 0.207
agi_english/1555 seed27712 0.207
agi_english/250 seed27712 0.207
agi_english/25 seed27712 0.207
agi_english/2258 seed27712 0.207
agi_english/235 seed27712 0.207
agi_english/1520 seed27712 0.207
agi_english/1306 seed27712 0.207
agi_english/1256 seed27712 0.207
agi_english/151 seed27712 0.207
agi_english/155 seed27712 0.207
agi_english/2475 seed24734 0.207
agi_english/2524 seed24734 0.207
agi_english/77 seed24734 0.207
agi_english/1793 seed24734 0.207
agi_english/240 seed24734 0.207
agi_english/2415 seed24734 0.207
agi_english/2409 seed24734 0.207
agi_english/2407 seed24734 0.207
agi_english/2429 seed24734 0.207
agi_english/2464 seed24734 0.207
agi_english/2444 seed24734 0.207
agi_english/2438 seed24734 0.207
agi_english/200 seed24734 0.207
agi_english/28 seed24734 0.207
agi_english/2349 seed24734 0.207
agi_english/2486 seed24734 0.207
agi_english/2248 seed24734 0.207
agi_english/221 seed24734 0.207
agi_english/2226 seed24734 0.207
agi_english/2500 seed24734 0.207
agi_english/2482 seed24734 0.207
agi_english/2361 seed24734 0.207
agi_english/2512 seed24734 0.207
agi_english/2509 seed24734 0.207
agi_english/3 seed24734 0.207
agi_english/30 seed24734 0.207
agi_english/2010 seed24734 0.207
agi_english/45 seed24734 0.207
agi_english/142 seed24734 0.207
agi_english/89 seed24734 0.207
agi_english/67 seed24734 0.207
agi_english/128 seed24734 0.207
agi_english/205 seed24734 0.207
agi_english/1862 seed24734 0.207
agi_english/191 seed24734 0.207
agi_english/1828 seed24734 0.207
agi_english/1144 seed16385 0.205
agi_english/1192 seed16385 0.205
agi_english/1215 seed16385 0.205
agi_english/1223 seed16385 0.205
agi_english/1835 seed16385 0.205
agi_english/1921 seed16385 0.205
agi_english/2327 seed16385 0.205
agi_english/1564 seed16385 0.205
agi_english/1490 seed16385 0.205
agi_english/1486 seed16385 0.205
agi_english/1472 seed16385 0.205
agi_english/1458 seed16385 0.205
agi_english/1443 seed16385 0.205
agi_english/1417 seed16385 0.205
agi_english/1398 seed16385 0.205
agi_english/1641 seed16385 0.205
agi_english/1568 seed16385 0.205
agi_english/1596 seed16385 0.205
agi_english/1595 seed16385 0.205
agi_english/1252 seed16385 0.205
agi_english/1313 seed16385 0.205
agi_english/1275 seed16385 0.205
agi_english/1369 seed16385 0.205
agi_english/1380 seed16385 0.205

Suspect problems

These are 10 problems with the lowest correlation with the overall evaluation (i.e. better models tend to do worse on these. )

example_link pass1_of_ex tau
agi_english/614 0.500 -0.745
agi_english/339 0.400 -0.730
agi_english/477 0.700 -0.618
agi_english/1883 0.300 -0.618
agi_english/1650 0.300 -0.618
agi_english/308 0.700 -0.618
agi_english/317 0.700 -0.618
agi_english/1201 0.300 -0.618
agi_english/581 0.700 -0.618
agi_english/1880 0.300 -0.618

Histogram of accuracies

Histogram of problems by the accuracy on each problem.

Histogram of difficulties

Histogram of problems by the minimum win rate to solve each problem.