agi_english_step27500: by examples

Results Paper Code


Not solved by any model

There are 653 examples not solved by any model. Solving some of these can be a good signal that your model is indeed better than leading models if these are good problems.
agi_english/1, agi_english/10, agi_english/1005, agi_english/1007, agi_english/1012, agi_english/1014, agi_english/1016, agi_english/1019, agi_english/1020, agi_english/1022, agi_english/1027, agi_english/1028, agi_english/103, agi_english/1034, agi_english/1039, agi_english/1043, agi_english/1046, agi_english/1049, agi_english/1054, agi_english/1057, agi_english/1061, agi_english/1062, agi_english/1064, agi_english/1065, agi_english/1069, agi_english/1071, agi_english/1072, agi_english/1076, agi_english/1078, agi_english/1081, agi_english/1083, agi_english/1085, agi_english/1086, agi_english/1088, agi_english/109, agi_english/1093, agi_english/1094, agi_english/1097, agi_english/1101, agi_english/1103, agi_english/1105, agi_english/1106, agi_english/1110, agi_english/1112, agi_english/1114, agi_english/1117, agi_english/1120, agi_english/1122, agi_english/1127, agi_english/1129, agi_english/1130, agi_english/1134, agi_english/1136, agi_english/1146, agi_english/1148, agi_english/1149, agi_english/116, agi_english/1160, agi_english/1161, agi_english/1163, agi_english/1165, agi_english/1169, agi_english/1177, agi_english/1178, agi_english/119, agi_english/1190, agi_english/1194, agi_english/1195, agi_english/120, agi_english/1204, agi_english/1206, agi_english/1207, agi_english/1208, agi_english/1209, agi_english/121, agi_english/1210, agi_english/1216, agi_english/1219, agi_english/1225, agi_english/1226, agi_english/1233, agi_english/1234, agi_english/1238, agi_english/1239, agi_english/124, agi_english/1244, agi_english/1245, agi_english/1247, agi_english/1254, agi_english/1255, agi_english/1258, agi_english/126, agi_english/1262, agi_english/1265, agi_english/1268, agi_english/1272, agi_english/1276, agi_english/1278, agi_english/1279, agi_english/128, agi_english/1283, agi_english/129, agi_english/1292, agi_english/1293, agi_english/1298, agi_english/1299, agi_english/1312, agi_english/1314, agi_english/1316, agi_english/1317, agi_english/132, agi_english/1321, agi_english/1323, agi_english/1327, agi_english/133, agi_english/1335, agi_english/1338, agi_english/1340, agi_english/1341, agi_english/1342, agi_english/1343, agi_english/1345, agi_english/1346, agi_english/1347, agi_english/1349, agi_english/135, agi_english/1351, agi_english/1352, agi_english/1357, agi_english/1360, agi_english/1362, agi_english/1363, agi_english/1367, agi_english/137, agi_english/1372, agi_english/1376, agi_english/1378, agi_english/1379, agi_english/1382, agi_english/1383, agi_english/1386, agi_english/1387, agi_english/139, agi_english/1396, agi_english/140, agi_english/1400, agi_english/1402, agi_english/1414, agi_english/1416, agi_english/1418, agi_english/1422, agi_english/1426, agi_english/143, agi_english/1431, agi_english/1441, agi_english/1442, agi_english/1443, agi_english/1445, agi_english/1447, agi_english/145, agi_english/1452, agi_english/146, agi_english/1460, agi_english/1461, agi_english/1462, agi_english/1466, agi_english/1468, agi_english/1473, agi_english/1481, agi_english/1482, agi_english/1483, agi_english/1491, agi_english/1492, agi_english/1493, agi_english/1498, agi_english/1499, agi_english/150, agi_english/1500, agi_english/1501, agi_english/1503, agi_english/1507, agi_english/1509, agi_english/1512, agi_english/1514, agi_english/1517, agi_english/1519, agi_english/1521, agi_english/1523, agi_english/1524, agi_english/1526, agi_english/1527, agi_english/153, agi_english/1531, agi_english/1532, agi_english/1538, agi_english/1540, agi_english/1541, agi_english/1544, agi_english/1547, agi_english/1548, agi_english/155, agi_english/1557, agi_english/156, agi_english/1560, agi_english/1565, agi_english/1566, agi_english/1570, agi_english/1572, agi_english/1573, agi_english/1578, agi_english/158, agi_english/1585, agi_english/1586, agi_english/1590, agi_english/1597, agi_english/1599, agi_english/160, agi_english/1605, agi_english/1609, agi_english/1610, agi_english/1614, agi_english/1618, agi_english/1625, agi_english/1627, agi_english/1628, agi_english/163, agi_english/1634, agi_english/1635, agi_english/1637, agi_english/1640, agi_english/1644, agi_english/1648, agi_english/1649, agi_english/1651, agi_english/1653, agi_english/1655, agi_english/1663, agi_english/1667, agi_english/1672, agi_english/1676, agi_english/1679, agi_english/1684, agi_english/1686, agi_english/1690, agi_english/1693, agi_english/1699, agi_english/17, agi_english/170, agi_english/1701, agi_english/1704, agi_english/1707, agi_english/1711, agi_english/1718, agi_english/1727, agi_english/1728, agi_english/1729, agi_english/1730, agi_english/1731, agi_english/1733, agi_english/1738, agi_english/174, agi_english/1740, agi_english/1741, agi_english/1742, agi_english/1743, agi_english/1745, agi_english/1746, agi_english/1748, agi_english/1751, agi_english/1752, agi_english/1753, agi_english/1754, agi_english/1757, agi_english/1758, agi_english/1759, agi_english/1765, agi_english/1766, agi_english/1767, agi_english/1768, agi_english/1769, agi_english/177, agi_english/1770, agi_english/1772, agi_english/1781, agi_english/1783, agi_english/1786, agi_english/1803, agi_english/1811, agi_english/1825, agi_english/1826, agi_english/183, agi_english/1838, agi_english/184, agi_english/1840, agi_english/1841, agi_english/1843, agi_english/1846, agi_english/185, agi_english/1850, agi_english/1856, agi_english/1857, agi_english/1858, agi_english/1865, agi_english/1866, agi_english/1871, agi_english/1874, agi_english/1876, agi_english/1877, agi_english/1880, agi_english/1881, agi_english/1883, agi_english/1886, agi_english/1889, agi_english/189, agi_english/1892, agi_english/1896, agi_english/1899, agi_english/190, agi_english/1901, agi_english/1904, agi_english/1906, agi_english/191, agi_english/1913, agi_english/1918, agi_english/192, agi_english/1928, agi_english/194, agi_english/1940, agi_english/1945, agi_english/2000, agi_english/2010, agi_english/2012, agi_english/2017, agi_english/2019, agi_english/2024, agi_english/2025, agi_english/2026, agi_english/2040, agi_english/2041, agi_english/2042, agi_english/2052, agi_english/2064, agi_english/2070, agi_english/2077, agi_english/208, agi_english/2083, agi_english/2084, agi_english/2092, agi_english/2093, agi_english/210, agi_english/2101, agi_english/2151, agi_english/2164, agi_english/2166, agi_english/2178, agi_english/218, agi_english/2185, agi_english/2187, agi_english/2199, agi_english/22, agi_english/2207, agi_english/2215, agi_english/2216, agi_english/222, agi_english/2225, agi_english/2229, agi_english/223, agi_english/225, agi_english/2268, agi_english/2292, agi_english/2296, agi_english/2299, agi_english/23, agi_english/2306, agi_english/2318, agi_english/2321, agi_english/2323, agi_english/2329, agi_english/2332, agi_english/2334, agi_english/2338, agi_english/2346, agi_english/2356, agi_english/2359, agi_english/2361, agi_english/2368, agi_english/2372, agi_english/2383, agi_english/239, agi_english/2390, agi_english/2396, agi_english/24, agi_english/2401, agi_english/2404, agi_english/2409, agi_english/2412, agi_english/2414, agi_english/2415, agi_english/2416, agi_english/2418, agi_english/2426, agi_english/2431, agi_english/2436, agi_english/2438, agi_english/2448, agi_english/245, agi_english/2452, agi_english/2456, agi_english/2458, agi_english/2459, agi_english/2462, agi_english/2466, agi_english/2468, agi_english/2469, agi_english/247, agi_english/2470, agi_english/248, agi_english/2483, agi_english/2489, agi_english/249, agi_english/2492, agi_english/2493, agi_english/2496, agi_english/2499, agi_english/250, agi_english/2501, agi_english/2505, agi_english/2514, agi_english/252, agi_english/2522, agi_english/2524, agi_english/2525, agi_english/2526, agi_english/2532, agi_english/2534, agi_english/2536, agi_english/2537, agi_english/2539, agi_english/2540, agi_english/256, agi_english/258, agi_english/263, agi_english/264, agi_english/267, agi_english/268, agi_english/269, agi_english/273, agi_english/274, agi_english/279, agi_english/280, agi_english/281, agi_english/283, agi_english/290, agi_english/291, agi_english/292, agi_english/294, agi_english/296, agi_english/298, agi_english/3, agi_english/301, agi_english/309, agi_english/313, agi_english/320, agi_english/327, agi_english/328, agi_english/33, agi_english/330, agi_english/335, agi_english/337, agi_english/34, agi_english/342, agi_english/343, agi_english/344, agi_english/358, agi_english/367, agi_english/374, agi_english/378, agi_english/381, agi_english/386, agi_english/388, agi_english/392, agi_english/396, agi_english/397, agi_english/402, agi_english/407, agi_english/408, agi_english/412, agi_english/413, agi_english/420, agi_english/423, agi_english/424, agi_english/434, agi_english/438, agi_english/44, agi_english/444, agi_english/446, agi_english/45, agi_english/452, agi_english/455, agi_english/459, agi_english/464, agi_english/471, agi_english/473, agi_english/48, agi_english/480, agi_english/488, agi_english/49, agi_english/490, agi_english/496, agi_english/497, agi_english/499, agi_english/501, agi_english/504, agi_english/508, agi_english/513, agi_english/515, agi_english/519, agi_english/52, agi_english/523, agi_english/532, agi_english/535, agi_english/538, agi_english/540, agi_english/541, agi_english/542, agi_english/548, agi_english/554, agi_english/555, agi_english/556, agi_english/557, agi_english/566, agi_english/568, agi_english/57, agi_english/571, agi_english/572, agi_english/576, agi_english/58, agi_english/590, agi_english/595, agi_english/60, agi_english/608, agi_english/609, agi_english/611, agi_english/613, agi_english/615, agi_english/617, agi_english/622, agi_english/626, agi_english/628, agi_english/632, agi_english/640, agi_english/642, agi_english/651, agi_english/652, agi_english/654, agi_english/657, agi_english/661, agi_english/692, agi_english/694, agi_english/696, agi_english/703, agi_english/706, agi_english/715, agi_english/716, agi_english/718, agi_english/719, agi_english/72, agi_english/720, agi_english/725, agi_english/73, agi_english/731, agi_english/732, agi_english/733, agi_english/738, agi_english/74, agi_english/740, agi_english/742, agi_english/746, agi_english/748, agi_english/753, agi_english/76, agi_english/760, agi_english/765, agi_english/766, agi_english/77, agi_english/771, agi_english/772, agi_english/773, agi_english/784, agi_english/79, agi_english/8, agi_english/80, agi_english/801, agi_english/803, agi_english/804, agi_english/807, agi_english/809, agi_english/811, agi_english/821, agi_english/829, agi_english/835, agi_english/845, agi_english/847, agi_english/849, agi_english/850, agi_english/853, agi_english/854, agi_english/859, agi_english/86, agi_english/861, agi_english/868, agi_english/869, agi_english/876, agi_english/881, agi_english/887, agi_english/891, agi_english/899, agi_english/90, agi_english/900, agi_english/901, agi_english/905, agi_english/907, agi_english/91, agi_english/910, agi_english/916, agi_english/920, agi_english/924, agi_english/926, agi_english/928, agi_english/935, agi_english/944, agi_english/948, agi_english/953, agi_english/954, agi_english/958, agi_english/962, agi_english/964, agi_english/966, agi_english/967, agi_english/968, agi_english/969, agi_english/97, agi_english/970, agi_english/973, agi_english/974, agi_english/977, agi_english/984, agi_english/986, agi_english/99, agi_english/995, agi_english/997, agi_english/998, agi_english/999

Problems solved by 1 model only

example_link model min_pass1_of_model
agi_english/2086 seed14309 0.243
agi_english/2036 seed14309 0.243
agi_english/2206 seed14309 0.243
agi_english/2129 seed14309 0.243
agi_english/2120 seed14309 0.243
agi_english/2234 seed14309 0.243
agi_english/2239 seed14309 0.243
agi_english/2209 seed14309 0.243
agi_english/573 seed14309 0.243
agi_english/53 seed14309 0.243
agi_english/528 seed14309 0.243
agi_english/2256 seed14309 0.243
agi_english/2242 seed14309 0.243
agi_english/806 seed14309 0.243
agi_english/834 seed14309 0.243
agi_english/882 seed14309 0.243
agi_english/873 seed14309 0.243
agi_english/1955 seed14309 0.243
agi_english/579 seed14309 0.243
agi_english/570 seed14309 0.243
agi_english/547 seed14309 0.243
agi_english/583 seed14309 0.243
agi_english/371 seed14309 0.243
agi_english/393 seed14309 0.243
agi_english/1166 seed14309 0.243
agi_english/1198 seed14309 0.243
agi_english/690 seed14309 0.243
agi_english/670 seed14309 0.243
agi_english/667 seed14309 0.243
agi_english/481 seed14309 0.243
agi_english/476 seed14309 0.243
agi_english/470 seed14309 0.243
agi_english/1139 seed14309 0.243
agi_english/1291 seed14309 0.243
agi_english/1261 seed14309 0.243
agi_english/1288 seed14309 0.243
agi_english/1284 seed14309 0.243
agi_english/1807 seed14309 0.243
agi_english/1819 seed14309 0.243
agi_english/1666 seed14309 0.243
agi_english/1834 seed14309 0.243
agi_english/345 seed14309 0.243
agi_english/325 seed14309 0.243
agi_english/1579 seed14309 0.243
agi_english/1563 seed14309 0.243
agi_english/40 seed14309 0.243
agi_english/350 seed14309 0.243
agi_english/31 seed14309 0.243
agi_english/303 seed14309 0.243
agi_english/841 seed14309 0.243
agi_english/892 seed14309 0.243
agi_english/2324 seed14309 0.243
agi_english/848 seed14309 0.243
agi_english/790 seed14309 0.243
agi_english/793 seed14309 0.243
agi_english/823 seed14309 0.243
agi_english/785 seed14309 0.243
agi_english/1189 seed14309 0.243
agi_english/118 seed14309 0.243
agi_english/1221 seed14309 0.243
agi_english/722 seed14309 0.243
agi_english/2310 seed14309 0.243
agi_english/832 seed14309 0.243
agi_english/2144 seed6115 0.242
agi_english/2189 seed6115 0.242
agi_english/2062 seed6115 0.242
agi_english/2217 seed6115 0.242
agi_english/2020 seed6115 0.242
agi_english/1203 seed6115 0.242
agi_english/1425 seed6115 0.242
agi_english/1175 seed6115 0.242
agi_english/1167 seed6115 0.242
agi_english/1170 seed6115 0.242
agi_english/2249 seed6115 0.242
agi_english/2188 seed6115 0.242
agi_english/2196 seed6115 0.242
agi_english/2279 seed6115 0.242
agi_english/1623 seed6115 0.242
agi_english/1620 seed6115 0.242
agi_english/1716 seed6115 0.242
agi_english/1703 seed6115 0.242
agi_english/1691 seed6115 0.242
agi_english/1668 seed6115 0.242
agi_english/1898 seed6115 0.242
agi_english/1902 seed6115 0.242
agi_english/1862 seed6115 0.242
agi_english/1870 seed6115 0.242
agi_english/1879 seed6115 0.242
agi_english/1814 seed6115 0.242
agi_english/1311 seed6115 0.242
agi_english/1318 seed6115 0.242
agi_english/1281 seed6115 0.242
agi_english/1842 seed6115 0.242
agi_english/1496 seed6115 0.242
agi_english/425 seed6115 0.242
agi_english/1484 seed6115 0.242
agi_english/1475 seed6115 0.242
agi_english/1361 seed6115 0.242
agi_english/1365 seed6115 0.242
agi_english/1487 seed6115 0.242
agi_english/1421 seed6115 0.242
agi_english/1147 seed6115 0.242
agi_english/1339 seed6115 0.242
agi_english/1408 seed6115 0.242
agi_english/2541 seed6115 0.242
agi_english/1607 seed6115 0.242
agi_english/1567 seed6115 0.242
agi_english/2461 seed6115 0.242
agi_english/2531 seed6115 0.242
agi_english/1785 seed6115 0.242
agi_english/1779 seed6115 0.242
agi_english/2314 seed6115 0.242
agi_english/2293 seed6115 0.242
agi_english/1737 seed6115 0.242
agi_english/1675 seed6115 0.242
agi_english/1735 seed6115 0.242
agi_english/1722 seed6115 0.242
agi_english/2520 seed6115 0.242
agi_english/361 seed6115 0.242
agi_english/419 seed6115 0.242
agi_english/2465 seed6115 0.242
agi_english/2490 seed6115 0.242
agi_english/2479 seed6115 0.242
agi_english/2506 seed6115 0.242
agi_english/2518 seed6115 0.242
agi_english/1582 seed6115 0.242
agi_english/1601 seed6115 0.242
agi_english/940 seed6115 0.242
agi_english/943 seed6115 0.242
agi_english/982 seed6115 0.242
agi_english/2328 seed6115 0.242
agi_english/1537 seed6115 0.242
agi_english/2432 seed6115 0.242
agi_english/2376 seed6115 0.242
agi_english/2394 seed6115 0.242
agi_english/2457 seed6115 0.242
agi_english/2442 seed6115 0.242
agi_english/2422 seed6115 0.242
agi_english/1237 seed6115 0.242
agi_english/1197 seed6115 0.242
agi_english/1021 seed6115 0.242
agi_english/996 seed149 0.241
agi_english/994 seed149 0.241
agi_english/992 seed149 0.241
agi_english/1004 seed149 0.241
agi_english/1013 seed149 0.241
agi_english/1010 seed149 0.241
agi_english/704 seed149 0.241
agi_english/824 seed149 0.241
agi_english/938 seed149 0.241
agi_english/921 seed149 0.241
agi_english/1954 seed149 0.241
agi_english/929 seed149 0.241
agi_english/2013 seed149 0.241
agi_english/1123 seed149 0.241
agi_english/1108 seed149 0.241
agi_english/1107 seed149 0.241
agi_english/1080 seed149 0.241
agi_english/1528 seed149 0.241
agi_english/1533 seed149 0.241
agi_english/1549 seed149 0.241
agi_english/2307 seed149 0.241
agi_english/1763 seed149 0.241
agi_english/370 seed149 0.241
agi_english/2460 seed149 0.241
agi_english/2447 seed149 0.241
agi_english/2053 seed149 0.241
agi_english/631 seed149 0.241
agi_english/663 seed149 0.241
agi_english/1932 seed149 0.241
agi_english/1915 seed149 0.241
agi_english/1822 seed149 0.241
agi_english/1925 seed149 0.241
agi_english/641 seed149 0.241
agi_english/567 seed149 0.241
agi_english/591 seed149 0.241
agi_english/2255 seed149 0.241
agi_english/2131 seed149 0.241
agi_english/2121 seed149 0.241
agi_english/2142 seed149 0.241
agi_english/2260 seed149 0.241
agi_english/2014 seed149 0.241
agi_english/1067 seed149 0.241
agi_english/1042 seed149 0.241
agi_english/1074 seed149 0.241
agi_english/1073 seed149 0.241
agi_english/1089 seed149 0.241
agi_english/1100 seed149 0.241
agi_english/1910 seed24762 0.234
agi_english/1079 seed24762 0.234
agi_english/988 seed24762 0.234
agi_english/1131 seed24762 0.234
agi_english/1053 seed24762 0.234
agi_english/1063 seed24762 0.234
agi_english/1820 seed24762 0.234
agi_english/1070 seed24762 0.234
agi_english/1099 seed24762 0.234
agi_english/1051 seed24762 0.234
agi_english/989 seed24762 0.234
agi_english/942 seed24762 0.234
agi_english/360 seed24762 0.234
agi_english/956 seed24762 0.234
agi_english/993 seed24762 0.234
agi_english/1685 seed30841 0.232
agi_english/171 seed30841 0.232
agi_english/1641 seed30841 0.232
agi_english/1647 seed30841 0.232
agi_english/186 seed30841 0.232
agi_english/1812 seed30841 0.232
agi_english/1657 seed30841 0.232
agi_english/1916 seed30841 0.232
agi_english/219 seed30841 0.232
agi_english/2028 seed30841 0.232
agi_english/106 seed30841 0.232
agi_english/1996 seed30841 0.232
agi_english/1326 seed30841 0.232
agi_english/151 seed30841 0.232
agi_english/1790 seed30841 0.232
agi_english/1923 seed30841 0.232
agi_english/1894 seed30841 0.232
agi_english/1719 seed30841 0.232
agi_english/232 seed30841 0.232
agi_english/2545 seed30841 0.232
agi_english/1791 seed30841 0.232
agi_english/1799 seed30841 0.232
agi_english/1958 seed30841 0.232
agi_english/172 seed30841 0.232
agi_english/839 seed17495 0.230
agi_english/825 seed17495 0.230
agi_english/761 seed17495 0.230
agi_english/865 seed17495 0.230
agi_english/780 seed17495 0.230
agi_english/743 seed17495 0.230
agi_english/909 seed17495 0.230
agi_english/906 seed17495 0.230
agi_english/878 seed17495 0.230
agi_english/898 seed17495 0.230
agi_english/284 seed17495 0.230
agi_english/311 seed17495 0.230
agi_english/257 seed17495 0.230
agi_english/277 seed17495 0.230
agi_english/176 seed17495 0.230
agi_english/113 seed17495 0.230
agi_english/111 seed17495 0.230
agi_english/2445 seed17495 0.230
agi_english/108 seed17495 0.230
agi_english/93 seed17495 0.230
agi_english/98 seed17495 0.230
agi_english/983 seed17495 0.230
agi_english/457 seed17495 0.230
agi_english/1917 seed17495 0.230
agi_english/447 seed17495 0.230
agi_english/445 seed17495 0.230
agi_english/634 seed17495 0.230
agi_english/607 seed17495 0.230
agi_english/668 seed17495 0.230
agi_english/114 seed17495 0.230
agi_english/38 seed17495 0.230
agi_english/380 seed17495 0.230
agi_english/449 seed17495 0.230
agi_english/451 seed17495 0.230
agi_english/436 seed17495 0.230
agi_english/463 seed17495 0.230
agi_english/551 seed17495 0.230
agi_english/521 seed17495 0.230
agi_english/2124 seed17495 0.230
agi_english/498 seed17495 0.230
agi_english/1988 seed17495 0.230
agi_english/1056 seed17495 0.230
agi_english/1031 seed27712 0.226
agi_english/1033 seed27712 0.226
agi_english/1066 seed27712 0.226
agi_english/1068 seed27712 0.226
agi_english/1128 seed27712 0.226
agi_english/871 seed27712 0.226
agi_english/1126 seed27712 0.226
agi_english/925 seed27712 0.226
agi_english/955 seed27712 0.226
agi_english/951 seed27712 0.226
agi_english/950 seed27712 0.226
agi_english/1025 seed27712 0.226
agi_english/2317 seed27712 0.226
agi_english/2301 seed27712 0.226
agi_english/2274 seed27712 0.226
agi_english/2148 seed27712 0.226
agi_english/2410 seed27712 0.226
agi_english/562 seed27712 0.226
agi_english/644 seed27712 0.226
agi_english/676 seed27712 0.226
agi_english/558 seed27712 0.226
agi_english/516 seed27712 0.226
agi_english/1006 seed27712 0.226
agi_english/828 seed27712 0.226
agi_english/493 seed27712 0.226
agi_english/491 seed27712 0.226
agi_english/1520 seed27712 0.226
agi_english/347 seed27712 0.226
agi_english/265 seed27712 0.226
agi_english/336 seed27712 0.226
agi_english/422 seed27712 0.226
agi_english/2500 seed27712 0.226
agi_english/638 seed27712 0.226
agi_english/763 seed27712 0.226
agi_english/796 seed27712 0.226
agi_english/774 seed27712 0.226
agi_english/808 seed27712 0.226
agi_english/1002 seed27712 0.226
agi_english/695 seed27712 0.226
agi_english/699 seed27712 0.226
agi_english/752 seed27712 0.226
agi_english/2366 seed16385 0.225
agi_english/2393 seed16385 0.225
agi_english/237 seed16385 0.225
agi_english/240 seed16385 0.225
agi_english/1554 seed16385 0.225
agi_english/1552 seed16385 0.225
agi_english/2351 seed16385 0.225
agi_english/1550 seed16385 0.225
agi_english/221 seed16385 0.225
agi_english/2288 seed16385 0.225
agi_english/234 seed16385 0.225
agi_english/175 seed16385 0.225
agi_english/1571 seed16385 0.225
agi_english/205 seed16385 0.225
agi_english/1295 seed16385 0.225
agi_english/598 seed16385 0.225
agi_english/1411 seed16385 0.225
agi_english/1310 seed16385 0.225
agi_english/1319 seed16385 0.225
agi_english/1328 seed16385 0.225
agi_english/1348 seed16385 0.225
agi_english/141 seed16385 0.225
agi_english/1424 seed16385 0.225
agi_english/1587 seed16385 0.225
agi_english/161 seed16385 0.225
agi_english/1583 seed16385 0.225
agi_english/2474 seed16385 0.225
agi_english/28 seed16385 0.225
agi_english/30 seed16385 0.225
agi_english/272 seed16385 0.225
agi_english/2542 seed16385 0.225
agi_english/228 seed16385 0.225
agi_english/179 seed16385 0.225
agi_english/1639 seed16385 0.225
agi_english/1939 seed16385 0.225
agi_english/220 seed16385 0.225
agi_english/216 seed16385 0.225
agi_english/213 seed16385 0.225
agi_english/202 seed16385 0.225
agi_english/2360 seed16385 0.225
agi_english/200 seed16385 0.225
agi_english/1975 seed16385 0.225
agi_english/1979 seed16385 0.225
agi_english/1968 seed16385 0.225
agi_english/1974 seed16385 0.225
agi_english/165 seed16385 0.225
agi_english/1914 seed16385 0.225
agi_english/2003 seed16385 0.225
agi_english/2043 seed16385 0.225
agi_english/2073 seed16385 0.225
agi_english/196 seed16385 0.225
agi_english/89 seed16385 0.225
agi_english/88 seed16385 0.225
agi_english/107 seed16385 0.225
agi_english/105 seed16385 0.225
agi_english/87 seed16385 0.225
agi_english/1251 seed16385 0.225
agi_english/1227 seed16385 0.225
agi_english/1179 seed16385 0.225
agi_english/1205 seed16385 0.225
agi_english/67 seed16385 0.225
agi_english/1510 seed16385 0.225
agi_english/1480 seed16385 0.225
agi_english/1419 seed16385 0.225
agi_english/1285 seed16385 0.225
agi_english/1154 seed16385 0.225
agi_english/1158 seed16385 0.225
agi_english/1222 seed16385 0.225
agi_english/1214 seed16385 0.225
agi_english/62 seed16385 0.225
agi_english/1145 seed16385 0.225
agi_english/368 seed16385 0.225
agi_english/1464 seed16385 0.225
agi_english/1474 seed16385 0.225
agi_english/1471 seed16385 0.225
agi_english/1495 seed16385 0.225
agi_english/1264 seed16385 0.225
agi_english/32 seed16385 0.225
agi_english/4 seed16385 0.225
agi_english/439 seed16385 0.225
agi_english/1428 seed16385 0.225
agi_english/1430 seed16385 0.225
agi_english/142 seed16385 0.225
agi_english/2484 seed16385 0.225
agi_english/100 seed24734 0.225
agi_english/83 seed24734 0.225
agi_english/81 seed24734 0.225
agi_english/92 seed24734 0.225
agi_english/918 seed24734 0.225
agi_english/9 seed24734 0.225
agi_english/94 seed24734 0.225
agi_english/195 seed24734 0.225
agi_english/971 seed24734 0.225
agi_english/11 seed24734 0.225
agi_english/110 seed24734 0.225
agi_english/199 seed24734 0.225
agi_english/0 seed24734 0.225
agi_english/71 seed24734 0.225
agi_english/770 seed24734 0.225
agi_english/6 seed24734 0.225
agi_english/64 seed24734 0.225
agi_english/148 seed24734 0.225
agi_english/78 seed24734 0.225
agi_english/35 seed24734 0.225
agi_english/1829 seed24734 0.225
agi_english/1813 seed24734 0.225
agi_english/1815 seed24734 0.225
agi_english/2118 seed24734 0.225
agi_english/2061 seed24734 0.225
agi_english/212 seed24734 0.225
agi_english/215 seed24734 0.225
agi_english/18 seed24734 0.225
agi_english/167 seed24734 0.225
agi_english/235 seed24734 0.225
agi_english/1774 seed24734 0.225
agi_english/243 seed24734 0.225
agi_english/246 seed24734 0.225
agi_english/214 seed24734 0.225
agi_english/226 seed24734 0.225
agi_english/15 seed24734 0.225
agi_english/251 seed24734 0.225
agi_english/16 seed24734 0.225
agi_english/29 seed24734 0.225
agi_english/41 seed24734 0.225
agi_english/349 seed24734 0.225
agi_english/37 seed24734 0.225
agi_english/253 seed24734 0.225
agi_english/47 seed24734 0.225
agi_english/1576 seed6529 0.209
agi_english/2023 seed6529 0.209
agi_english/2089 seed6529 0.209
agi_english/1921 seed6529 0.209
agi_english/1873 seed6529 0.209
agi_english/1832 seed6529 0.209
agi_english/1942 seed6529 0.209
agi_english/1972 seed6529 0.209
agi_english/1574 seed6529 0.209
agi_english/1559 seed6529 0.209
agi_english/1535 seed6529 0.209
agi_english/1525 seed6529 0.209
agi_english/1788 seed6529 0.209
agi_english/1687 seed6529 0.209
agi_english/2339 seed6529 0.209
agi_english/1787 seed6529 0.209
agi_english/2127 seed6529 0.209
agi_english/1636 seed6529 0.209
agi_english/1632 seed6529 0.209
agi_english/1589 seed6529 0.209
agi_english/1633 seed6529 0.209
agi_english/1617 seed6529 0.209
agi_english/1630 seed6529 0.209
agi_english/1263 seed6529 0.209
agi_english/1260 seed6529 0.209
agi_english/1505 seed6529 0.209
agi_english/1511 seed6529 0.209
agi_english/1455 seed6529 0.209
agi_english/1488 seed6529 0.209
agi_english/1449 seed6529 0.209
agi_english/1450 seed6529 0.209
agi_english/1432 seed6529 0.209
agi_english/1412 seed6529 0.209
agi_english/1423 seed6529 0.209
agi_english/1427 seed6529 0.209
agi_english/1435 seed6529 0.209
agi_english/1616 seed6529 0.209
agi_english/1371 seed6529 0.209
agi_english/1405 seed6529 0.209
agi_english/1153 seed6529 0.209
agi_english/1152 seed6529 0.209
agi_english/1135 seed6529 0.209
agi_english/1368 seed6529 0.209
agi_english/1377 seed6529 0.209
agi_english/1394 seed6529 0.209
agi_english/1401 seed6529 0.209
agi_english/1315 seed6529 0.209
agi_english/1337 seed6529 0.209
agi_english/1329 seed6529 0.209
agi_english/1242 seed6529 0.209
agi_english/1217 seed6529 0.209
agi_english/1173 seed6529 0.209
agi_english/1159 seed6529 0.209
agi_english/1176 seed6529 0.209
agi_english/1200 seed6529 0.209

Suspect problems

These are 10 problems with the lowest correlation with the overall evaluation (i.e. better models tend to do worse on these. )

example_link pass1_of_ex tau
agi_english/739 0.500 -0.745
agi_english/693 0.600 -0.669
agi_english/1023 0.500 -0.626
agi_english/1824 0.300 -0.618
agi_english/410 0.400 -0.609
agi_english/323 0.600 -0.609
agi_english/896 0.600 -0.609
agi_english/976 0.400 -0.609
agi_english/683 0.600 -0.609
agi_english/1935 0.700 -0.553

Histogram of accuracies

Histogram of problems by the accuracy on each problem.

Histogram of difficulties

Histogram of problems by the minimum win rate to solve each problem.