agi_english_step50000: by examples

Results Paper Code


Not solved by any model

There are 659 examples not solved by any model. Solving some of these can be a good signal that your model is indeed better than leading models if these are good problems.
agi_english/1, agi_english/1000, agi_english/1004, agi_english/1010, agi_english/1012, agi_english/1013, agi_english/1019, agi_english/1020, agi_english/1021, agi_english/1027, agi_english/103, agi_english/1039, agi_english/1049, agi_english/1051, agi_english/1054, agi_english/1057, agi_english/1059, agi_english/1061, agi_english/1062, agi_english/1064, agi_english/1067, agi_english/1069, agi_english/107, agi_english/1076, agi_english/1078, agi_english/1080, agi_english/1083, agi_english/1085, agi_english/1087, agi_english/1088, agi_english/1089, agi_english/1093, agi_english/1094, agi_english/1099, agi_english/1100, agi_english/1101, agi_english/1105, agi_english/1107, agi_english/1108, agi_english/111, agi_english/1110, agi_english/1112, agi_english/1114, agi_english/1117, agi_english/1123, agi_english/1130, agi_english/1135, agi_english/1136, agi_english/1145, agi_english/1146, agi_english/1148, agi_english/1152, agi_english/1153, agi_english/1158, agi_english/1159, agi_english/1169, agi_english/1170, agi_english/1173, agi_english/1175, agi_english/1176, agi_english/1177, agi_english/1178, agi_english/1183, agi_english/1189, agi_english/119, agi_english/1190, agi_english/1194, agi_english/1195, agi_english/120, agi_english/1200, agi_english/1203, agi_english/1204, agi_english/1205, agi_english/1207, agi_english/1208, agi_english/1209, agi_english/1214, agi_english/1216, agi_english/1217, agi_english/1221, agi_english/1224, agi_english/1226, agi_english/1227, agi_english/1233, agi_english/1234, agi_english/1236, agi_english/1237, agi_english/1238, agi_english/124, agi_english/1242, agi_english/1243, agi_english/1244, agi_english/1247, agi_english/1251, agi_english/1257, agi_english/1258, agi_english/126, agi_english/1260, agi_english/1263, agi_english/1264, agi_english/1272, agi_english/1273, agi_english/1276, agi_english/1279, agi_english/128, agi_english/1281, agi_english/1284, agi_english/1285, agi_english/1287, agi_english/1288, agi_english/1291, agi_english/1292, agi_english/1293, agi_english/1298, agi_english/1299, agi_english/1310, agi_english/1311, agi_english/1312, agi_english/1315, agi_english/1316, agi_english/1318, agi_english/1326, agi_english/1327, agi_english/1329, agi_english/133, agi_english/1335, agi_english/1337, agi_english/1338, agi_english/1339, agi_english/1341, agi_english/1342, agi_english/1343, agi_english/1345, agi_english/1346, agi_english/1347, agi_english/1348, agi_english/1349, agi_english/135, agi_english/1352, agi_english/1357, agi_english/1359, agi_english/1360, agi_english/1361, agi_english/1362, agi_english/1365, agi_english/1368, agi_english/137, agi_english/1371, agi_english/1372, agi_english/1374, agi_english/1376, agi_english/1377, agi_english/1378, agi_english/1379, agi_english/1382, agi_english/1383, agi_english/1387, agi_english/1388, agi_english/1389, agi_english/139, agi_english/1394, agi_english/1396, agi_english/1397, agi_english/140, agi_english/1400, agi_english/1401, agi_english/1405, agi_english/1411, agi_english/1412, agi_english/1413, agi_english/1418, agi_english/1421, agi_english/1422, agi_english/1423, agi_english/1424, agi_english/1425, agi_english/1427, agi_english/1428, agi_english/143, agi_english/1431, agi_english/1432, agi_english/1435, agi_english/1441, agi_english/1442, agi_english/1444, agi_english/1445, agi_english/1447, agi_english/1449, agi_english/145, agi_english/1452, agi_english/1455, agi_english/1460, agi_english/1461, agi_english/1462, agi_english/1464, agi_english/1466, agi_english/1468, agi_english/1473, agi_english/1481, agi_english/1482, agi_english/1483, agi_english/1484, agi_english/1487, agi_english/1488, agi_english/149, agi_english/1495, agi_english/1496, agi_english/1498, agi_english/1499, agi_english/1500, agi_english/1503, agi_english/1505, agi_english/1507, agi_english/1509, agi_english/151, agi_english/1510, agi_english/1511, agi_english/1512, agi_english/1514, agi_english/1517, agi_english/1518, agi_english/1519, agi_english/1523, agi_english/1524, agi_english/1525, agi_english/1527, agi_english/153, agi_english/1532, agi_english/1535, agi_english/1537, agi_english/1538, agi_english/1540, agi_english/1541, agi_english/1542, agi_english/1544, agi_english/155, agi_english/1552, agi_english/1554, agi_english/1557, agi_english/1559, agi_english/156, agi_english/1560, agi_english/1561, agi_english/1563, agi_english/1565, agi_english/1566, agi_english/1567, agi_english/1571, agi_english/1572, agi_english/1573, agi_english/1574, agi_english/1576, agi_english/1578, agi_english/158, agi_english/1586, agi_english/1589, agi_english/1591, agi_english/1597, agi_english/1601, agi_english/1605, agi_english/1607, agi_english/1608, agi_english/1610, agi_english/1616, agi_english/1617, agi_english/1618, agi_english/1620, agi_english/1623, agi_english/1627, agi_english/1628, agi_english/1630, agi_english/1632, agi_english/1633, agi_english/1636, agi_english/1639, agi_english/1642, agi_english/1646, agi_english/1651, agi_english/1653, agi_english/1654, agi_english/1660, agi_english/1667, agi_english/1668, agi_english/1672, agi_english/1676, agi_english/1677, agi_english/1679, agi_english/1680, agi_english/1682, agi_english/1684, agi_english/1686, agi_english/1687, agi_english/1699, agi_english/170, agi_english/1700, agi_english/1701, agi_english/1703, agi_english/1706, agi_english/1707, agi_english/171, agi_english/1716, agi_english/172, agi_english/1721, agi_english/1722, agi_english/1727, agi_english/1731, agi_english/1733, agi_english/1737, agi_english/1740, agi_english/1741, agi_english/1742, agi_english/1743, agi_english/1745, agi_english/1746, agi_english/1748, agi_english/1751, agi_english/1752, agi_english/1753, agi_english/1754, agi_english/1757, agi_english/1758, agi_english/1759, agi_english/1765, agi_english/1766, agi_english/1767, agi_english/1768, agi_english/1769, agi_english/177, agi_english/1770, agi_english/1772, agi_english/1778, agi_english/1781, agi_english/1782, agi_english/1783, agi_english/1785, agi_english/1788, agi_english/1789, agi_english/1794, agi_english/1796, agi_english/1799, agi_english/1800, agi_english/1802, agi_english/1804, agi_english/1810, agi_english/1813, agi_english/1814, agi_english/1815, agi_english/1816, agi_english/1822, agi_english/1824, agi_english/1825, agi_english/1830, agi_english/1834, agi_english/184, agi_english/1840, agi_english/1841, agi_english/1843, agi_english/1846, agi_english/1847, agi_english/1848, agi_english/185, agi_english/1850, agi_english/1856, agi_english/1858, agi_english/1860, agi_english/1866, agi_english/1868, agi_english/1871, agi_english/1873, agi_english/1874, agi_english/1878, agi_english/1881, agi_english/1889, agi_english/189, agi_english/1891, agi_english/1892, agi_english/1893, agi_english/1898, agi_english/1899, agi_english/190, agi_english/1901, agi_english/1902, agi_english/1903, agi_english/1906, agi_english/191, agi_english/1917, agi_english/1918, agi_english/1933, agi_english/1935, agi_english/1936, agi_english/1937, agi_english/1942, agi_english/1943, agi_english/1961, agi_english/1964, agi_english/1981, agi_english/1982, agi_english/1983, agi_english/1991, agi_english/1992, agi_english/2002, agi_english/2007, agi_english/2012, agi_english/2017, agi_english/2023, agi_english/2024, agi_english/2025, agi_english/2040, agi_english/2041, agi_english/2042, agi_english/2044, agi_english/2061, agi_english/2069, agi_english/2077, agi_english/208, agi_english/2080, agi_english/2081, agi_english/2083, agi_english/210, agi_english/2101, agi_english/2104, agi_english/2110, agi_english/2113, agi_english/2116, agi_english/2117, agi_english/2123, agi_english/2124, agi_english/2125, agi_english/2126, agi_english/2139, agi_english/2141, agi_english/2142, agi_english/2144, agi_english/2148, agi_english/2149, agi_english/2156, agi_english/2166, agi_english/2170, agi_english/2178, agi_english/218, agi_english/2187, agi_english/2188, agi_english/2189, agi_english/219, agi_english/2191, agi_english/2196, agi_english/2198, agi_english/22, agi_english/2208, agi_english/2210, agi_english/2213, agi_english/2214, agi_english/2217, agi_english/222, agi_english/2223, agi_english/2225, agi_english/2227, agi_english/2229, agi_english/223, agi_english/2230, agi_english/2235, agi_english/2236, agi_english/2241, agi_english/2243, agi_english/2246, agi_english/2249, agi_english/2251, agi_english/2255, agi_english/2262, agi_english/2266, agi_english/2274, agi_english/2275, agi_english/2279, agi_english/2280, agi_english/2287, agi_english/2293, agi_english/2294, agi_english/2296, agi_english/2297, agi_english/2299, agi_english/230, agi_english/2301, agi_english/2304, agi_english/2306, agi_english/2314, agi_english/2317, agi_english/232, agi_english/2320, agi_english/2321, agi_english/2323, agi_english/2328, agi_english/2330, agi_english/2333, agi_english/2340, agi_english/2341, agi_english/2342, agi_english/2348, agi_english/2349, agi_english/2353, agi_english/2355, agi_english/2357, agi_english/2359, agi_english/2360, agi_english/2365, agi_english/2369, agi_english/237, agi_english/2370, agi_english/2373, agi_english/2376, agi_english/2378, agi_english/2380, agi_english/2381, agi_english/2390, agi_english/2394, agi_english/2395, agi_english/2396, agi_english/24, agi_english/2401, agi_english/2405, agi_english/2408, agi_english/2409, agi_english/2418, agi_english/2421, agi_english/2422, agi_english/2427, agi_english/2432, agi_english/2436, agi_english/2442, agi_english/2445, agi_english/2449, agi_english/245, agi_english/2450, agi_english/2452, agi_english/2457, agi_english/2460, agi_english/2461, agi_english/2465, agi_english/247, agi_english/2478, agi_english/2479, agi_english/248, agi_english/2485, agi_english/2488, agi_english/249, agi_english/2490, agi_english/2497, agi_english/250, agi_english/2504, agi_english/2505, agi_english/2506, agi_english/2507, agi_english/2511, agi_english/2513, agi_english/2518, agi_english/2519, agi_english/2520, agi_english/2521, agi_english/2533, agi_english/2534, agi_english/2535, agi_english/2541, agi_english/2542, agi_english/2543, agi_english/2545, agi_english/273, agi_english/283, agi_english/298, agi_english/311, agi_english/313, agi_english/320, agi_english/33, agi_english/344, agi_english/347, agi_english/349, agi_english/365, agi_english/367, agi_english/369, agi_english/38, agi_english/405, agi_english/424, agi_english/431, agi_english/435, agi_english/445, agi_english/45, agi_english/453, agi_english/473, agi_english/488, agi_english/498, agi_english/499, agi_english/504, agi_english/521, agi_english/532, agi_english/535, agi_english/540, agi_english/552, agi_english/556, agi_english/576, agi_english/58, agi_english/580, agi_english/586, agi_english/60, agi_english/613, agi_english/644, agi_english/648, agi_english/650, agi_english/670, agi_english/680, agi_english/692, agi_english/699, agi_english/718, agi_english/73, agi_english/732, agi_english/74, agi_english/740, agi_english/742, agi_english/76, agi_english/766, agi_english/77, agi_english/778, agi_english/798, agi_english/8, agi_english/800, agi_english/811, agi_english/817, agi_english/853, agi_english/855, agi_english/86, agi_english/901, agi_english/905, agi_english/908, agi_english/911, agi_english/919, agi_english/920, agi_english/921, agi_english/929, agi_english/936, agi_english/938, agi_english/939, agi_english/940, agi_english/943, agi_english/948, agi_english/954, agi_english/958, agi_english/962, agi_english/967, agi_english/97, agi_english/970, agi_english/972, agi_english/974, agi_english/977, agi_english/979, agi_english/982, agi_english/986, agi_english/988, agi_english/99, agi_english/994, agi_english/996, agi_english/997, agi_english/998

Problems solved by 1 model only

example_link model min_pass1_of_model
agi_english/753 seed17495 0.247
agi_english/761 seed17495 0.247
agi_english/780 seed17495 0.247
agi_english/725 seed17495 0.247
agi_english/696 seed17495 0.247
agi_english/748 seed17495 0.247
agi_english/676 seed17495 0.247
agi_english/652 seed17495 0.247
agi_english/651 seed17495 0.247
agi_english/678 seed17495 0.247
agi_english/605 seed17495 0.247
agi_english/463 seed17495 0.247
agi_english/46 seed17495 0.247
agi_english/436 seed17495 0.247
agi_english/513 seed17495 0.247
agi_english/501 seed17495 0.247
agi_english/428 seed17495 0.247
agi_english/1450 seed17495 0.247
agi_english/1443 seed17495 0.247
agi_english/332 seed17495 0.247
agi_english/261 seed17495 0.247
agi_english/269 seed17495 0.247
agi_english/327 seed17495 0.247
agi_english/393 seed17495 0.247
agi_english/383 seed17495 0.247
agi_english/434 seed17495 0.247
agi_english/2463 seed17495 0.247
agi_english/2453 seed17495 0.247
agi_english/2435 seed17495 0.247
agi_english/2539 seed17495 0.247
agi_english/186 seed17495 0.247
agi_english/1959 seed17495 0.247
agi_english/2143 seed17495 0.247
agi_english/2165 seed17495 0.247
agi_english/1775 seed17495 0.247
agi_english/2082 seed17495 0.247
agi_english/1717 seed17495 0.247
agi_english/1985 seed17495 0.247
agi_english/2064 seed17495 0.247
agi_english/1669 seed17495 0.247
agi_english/202 seed17495 0.247
agi_english/1988 seed17495 0.247
agi_english/1966 seed17495 0.247
agi_english/849 seed17495 0.247
agi_english/1926 seed17495 0.247
agi_english/2286 seed17495 0.247
agi_english/2250 seed17495 0.247
agi_english/2194 seed17495 0.247
agi_english/2446 seed17495 0.247
agi_english/2337 seed17495 0.247
agi_english/239 seed17495 0.247
agi_english/2391 seed17495 0.247
agi_english/2411 seed17495 0.247
agi_english/2413 seed17495 0.247
agi_english/2309 seed17495 0.247
agi_english/2272 seed17495 0.247
agi_english/2197 seed17495 0.247
agi_english/2433 seed17495 0.247
agi_english/291 seed17495 0.247
agi_english/281 seed17495 0.247
agi_english/72 seed16385 0.243
agi_english/1262 seed16385 0.243
agi_english/1154 seed16385 0.243
agi_english/1265 seed16385 0.243
agi_english/28 seed16385 0.243
agi_english/321 seed16385 0.243
agi_english/342 seed16385 0.243
agi_english/32 seed16385 0.243
agi_english/34 seed16385 0.243
agi_english/1319 seed16385 0.243
agi_english/132 seed16385 0.243
agi_english/1351 seed16385 0.243
agi_english/146 seed16385 0.243
agi_english/141 seed16385 0.243
agi_english/44 seed16385 0.243
agi_english/441 seed16385 0.243
agi_english/4 seed16385 0.243
agi_english/1474 seed16385 0.243
agi_english/1493 seed16385 0.243
agi_english/142 seed16385 0.243
agi_english/411 seed16385 0.243
agi_english/1261 seed16385 0.243
agi_english/1254 seed16385 0.243
agi_english/1225 seed16385 0.243
agi_english/1304 seed16385 0.243
agi_english/656 seed16385 0.243
agi_english/67 seed16385 0.243
agi_english/1163 seed16385 0.243
agi_english/640 seed16385 0.243
agi_english/62 seed16385 0.243
agi_english/585 seed16385 0.243
agi_english/607 seed16385 0.243
agi_english/52 seed16385 0.243
agi_english/558 seed16385 0.243
agi_english/568 seed16385 0.243
agi_english/49 seed16385 0.243
agi_english/746 seed16385 0.243
agi_english/80 seed16385 0.243
agi_english/220 seed16385 0.243
agi_english/2415 seed16385 0.243
agi_english/1546 seed16385 0.243
agi_english/2231 seed16385 0.243
agi_english/225 seed16385 0.243
agi_english/252 seed16385 0.243
agi_english/1625 seed16385 0.243
agi_english/161 seed16385 0.243
agi_english/165 seed16385 0.243
agi_english/1635 seed16385 0.243
agi_english/1570 seed16385 0.243
agi_english/150 seed16385 0.243
agi_english/1583 seed16385 0.243
agi_english/221 seed16385 0.243
agi_english/205 seed16385 0.243
agi_english/2011 seed16385 0.243
agi_english/200 seed16385 0.243
agi_english/2078 seed16385 0.243
agi_english/216 seed16385 0.243
agi_english/196 seed16385 0.243
agi_english/192 seed16385 0.243
agi_english/194 seed16385 0.243
agi_english/1859 seed16385 0.243
agi_english/1806 seed16385 0.243
agi_english/89 seed16385 0.243
agi_english/88 seed16385 0.243
agi_english/898 seed16385 0.243
agi_english/87 seed16385 0.243
agi_english/109 seed16385 0.243
agi_english/1047 seed16385 0.243
agi_english/679 seed149 0.241
agi_english/1529 seed149 0.241
agi_english/597 seed149 0.241
agi_english/658 seed149 0.241
agi_english/567 seed149 0.241
agi_english/587 seed149 0.241
agi_english/526 seed149 0.241
agi_english/455 seed149 0.241
agi_english/833 seed149 0.241
agi_english/300 seed149 0.241
agi_english/2522 seed149 0.241
agi_english/1547 seed149 0.241
agi_english/1851 seed149 0.241
agi_english/1829 seed149 0.241
agi_english/357 seed149 0.241
agi_english/449 seed149 0.241
agi_english/429 seed149 0.241
agi_english/2073 seed149 0.241
agi_english/747 seed149 0.241
agi_english/2339 seed149 0.241
agi_english/2318 seed149 0.241
agi_english/2043 seed149 0.241
agi_english/2290 seed149 0.241
agi_english/1229 seed149 0.241
agi_english/707 seed149 0.241
agi_english/1693 seed149 0.241
agi_english/1710 seed149 0.241
agi_english/2129 seed27712 0.237
agi_english/422 seed27712 0.237
agi_english/260 seed27712 0.237
agi_english/318 seed27712 0.237
agi_english/1879 seed27712 0.237
agi_english/1969 seed27712 0.237
agi_english/1634 seed27712 0.237
agi_english/284 seed27712 0.237
agi_english/2070 seed27712 0.237
agi_english/1675 seed27712 0.237
agi_english/743 seed27712 0.237
agi_english/785 seed27712 0.237
agi_english/1736 seed27712 0.237
agi_english/2114 seed27712 0.237
agi_english/713 seed27712 0.237
agi_english/1071 seed27712 0.237
agi_english/1222 seed27712 0.237
agi_english/2087 seed27712 0.237
agi_english/2253 seed27712 0.237
agi_english/2218 seed27712 0.237
agi_english/2206 seed27712 0.237
agi_english/2090 seed27712 0.237
agi_english/1720 seed27712 0.237
agi_english/1520 seed27712 0.237
agi_english/2336 seed27712 0.237
agi_english/2175 seed27712 0.237
agi_english/2325 seed27712 0.237
agi_english/2310 seed27712 0.237
agi_english/1323 seed27712 0.237
agi_english/326 seed27712 0.237
agi_english/562 seed27712 0.237
agi_english/457 seed27712 0.237
agi_english/491 seed27712 0.237
agi_english/406 seed27712 0.237
agi_english/1792 seed27712 0.237
agi_english/59 seed27712 0.237
agi_english/611 seed27712 0.237
agi_english/1133 seed27712 0.237
agi_english/1728 seed27712 0.237
agi_english/714 seed27712 0.237
agi_english/1295 seed27712 0.237
agi_english/2254 seed27712 0.237
agi_english/1016 seed14309 0.237
agi_english/2112 seed14309 0.237
agi_english/751 seed14309 0.237
agi_english/2420 seed14309 0.237
agi_english/1685 seed14309 0.237
agi_english/1729 seed14309 0.237
agi_english/2481 seed14309 0.237
agi_english/2059 seed14309 0.237
agi_english/1666 seed14309 0.237
agi_english/1657 seed14309 0.237
agi_english/2008 seed14309 0.237
agi_english/1723 seed14309 0.237
agi_english/812 seed14309 0.237
agi_english/426 seed14309 0.237
agi_english/895 seed14309 0.237
agi_english/373 seed14309 0.237
agi_english/563 seed14309 0.237
agi_english/1780 seed14309 0.237
agi_english/583 seed14309 0.237
agi_english/903 seed14309 0.237
agi_english/2528 seed14309 0.237
agi_english/1807 seed14309 0.237
agi_english/1890 seed14309 0.237
agi_english/1883 seed14309 0.237
agi_english/370 seed14309 0.237
agi_english/293 seed14309 0.237
agi_english/698 seed14309 0.237
agi_english/2119 seed14309 0.237
agi_english/2084 seed14309 0.237
agi_english/1219 seed24734 0.236
agi_english/296 seed24734 0.236
agi_english/337 seed24734 0.236
agi_english/1967 seed24734 0.236
agi_english/1865 seed24734 0.236
agi_english/1857 seed24734 0.236
agi_english/1838 seed24734 0.236
agi_english/371 seed24734 0.236
agi_english/1896 seed24734 0.236
agi_english/657 seed24734 0.236
agi_english/528 seed24734 0.236
agi_english/572 seed24734 0.236
agi_english/980 seed24734 0.236
agi_english/841 seed24734 0.236
agi_english/476 seed24734 0.236
agi_english/617 seed24734 0.236
agi_english/1340 seed24734 0.236
agi_english/309 seed24734 0.236
agi_english/2473 seed24734 0.236
agi_english/770 seed24734 0.236
agi_english/779 seed24734 0.236
agi_english/738 seed24734 0.236
agi_english/806 seed24734 0.236
agi_english/1116 seed24734 0.236
agi_english/715 seed24734 0.236
agi_english/1092 seed24734 0.236
agi_english/829 seed24734 0.236
agi_english/1038 seed24734 0.236
agi_english/1055 seed24734 0.236
agi_english/1001 seed24734 0.236
agi_english/1017 seed24734 0.236
agi_english/13 seed6115 0.233
agi_english/775 seed6115 0.233
agi_english/831 seed6115 0.233
agi_english/2054 seed6115 0.233
agi_english/1777 seed6115 0.233
agi_english/816 seed6115 0.233
agi_english/805 seed6115 0.233
agi_english/791 seed6115 0.233
agi_english/2062 seed6115 0.233
agi_english/1950 seed6115 0.233
agi_english/9 seed6115 0.233
agi_english/3 seed6115 0.233
agi_english/1821 seed6115 0.233
agi_english/1363 seed6115 0.233
agi_english/1386 seed6115 0.233
agi_english/1812 seed6115 0.233
agi_english/1912 seed6115 0.233
agi_english/90 seed6115 0.233
agi_english/1801 seed6115 0.233
agi_english/1747 seed30841 0.231
agi_english/1691 seed30841 0.231
agi_english/1696 seed30841 0.231
agi_english/1690 seed30841 0.231
agi_english/1704 seed30841 0.231
agi_english/824 seed30841 0.231
agi_english/1314 seed30841 0.231
agi_english/1283 seed30841 0.231
agi_english/1005 seed30841 0.231
agi_english/1065 seed30841 0.231
agi_english/1043 seed30841 0.231
agi_english/1073 seed30841 0.231
agi_english/1072 seed30841 0.231
agi_english/2456 seed30841 0.231
agi_english/2467 seed30841 0.231
agi_english/2439 seed30841 0.231
agi_english/243 seed30841 0.231
agi_english/2382 seed30841 0.231
agi_english/1356 seed30841 0.231
agi_english/2530 seed30841 0.231
agi_english/1649 seed30841 0.231
agi_english/1599 seed30841 0.231
agi_english/1590 seed30841 0.231
agi_english/1585 seed30841 0.231
agi_english/1587 seed30841 0.231
agi_english/1640 seed30841 0.231
agi_english/1644 seed30841 0.231
agi_english/1637 seed30841 0.231
agi_english/1648 seed30841 0.231
agi_english/1718 seed30841 0.231
agi_english/2417 seed30841 0.231
agi_english/1712 seed30841 0.231
agi_english/1528 seed30841 0.231
agi_english/1526 seed30841 0.231
agi_english/2476 seed30841 0.231
agi_english/1655 seed30841 0.231
agi_english/2284 seed30841 0.231
agi_english/1735 seed30841 0.231
agi_english/1730 seed30841 0.231
agi_english/1278 seed30841 0.231
agi_english/1255 seed30841 0.231
agi_english/1245 seed30841 0.231
agi_english/1853 seed30841 0.231
agi_english/1870 seed30841 0.231
agi_english/1886 seed30841 0.231
agi_english/1122 seed30841 0.231
agi_english/621 seed30841 0.231
agi_english/1161 seed30841 0.231
agi_english/1165 seed30841 0.231
agi_english/425 seed30841 0.231
agi_english/926 seed30841 0.231
agi_english/1149 seed30841 0.231
agi_english/1141 seed30841 0.231
agi_english/1206 seed30841 0.231
agi_english/1197 seed30841 0.231
agi_english/1139 seed30841 0.231
agi_english/1167 seed30841 0.231
agi_english/1791 seed30841 0.231
agi_english/992 seed30841 0.231
agi_english/964 seed30841 0.231
agi_english/1426 seed30841 0.231
agi_english/1494 seed30841 0.231
agi_english/1416 seed30841 0.231
agi_english/1475 seed30841 0.231
agi_english/1480 seed30841 0.231
agi_english/1438 seed30841 0.231
agi_english/1430 seed30841 0.231
agi_english/1429 seed30841 0.231
agi_english/1467 seed30841 0.231
agi_english/1414 seed30841 0.231
agi_english/1913 seed30841 0.231
agi_english/1408 seed30841 0.231
agi_english/1522 seed30841 0.231
agi_english/1582 seed30841 0.231
agi_english/1581 seed30841 0.231
agi_english/1579 seed30841 0.231
agi_english/1531 seed30841 0.231
agi_english/1919 seed30841 0.231
agi_english/1367 seed30841 0.231
agi_english/1402 seed30841 0.231
agi_english/1852 seed30841 0.231
agi_english/1867 seed30841 0.231
agi_english/1842 seed30841 0.231
agi_english/2524 seed30841 0.231
agi_english/2503 seed30841 0.231
agi_english/1395 seed30841 0.231
agi_english/1492 seed30841 0.231
agi_english/1491 seed30841 0.231
agi_english/1328 seed30841 0.231
agi_english/1849 seed30841 0.231
agi_english/913 seed30841 0.231
agi_english/1040 seed24762 0.225
agi_english/1037 seed24762 0.225
agi_english/1014 seed24762 0.225
agi_english/1121 seed24762 0.225
agi_english/1070 seed24762 0.225
agi_english/1053 seed24762 0.225
agi_english/1052 seed24762 0.225
agi_english/1041 seed24762 0.225
agi_english/1058 seed24762 0.225
agi_english/1063 seed24762 0.225
agi_english/1096 seed24762 0.225
agi_english/1091 seed24762 0.225
agi_english/1111 seed24762 0.225
agi_english/1115 seed24762 0.225
agi_english/2172 seed24762 0.225
agi_english/2053 seed24762 0.225
agi_english/2060 seed24762 0.225
agi_english/2326 seed24762 0.225
agi_english/1131 seed24762 0.225
agi_english/1132 seed24762 0.225
agi_english/991 seed24762 0.225
agi_english/1786 seed24762 0.225
agi_english/981 seed24762 0.225
agi_english/978 seed24762 0.225
agi_english/959 seed24762 0.225
agi_english/927 seed24762 0.225
agi_english/868 seed24762 0.225
agi_english/912 seed24762 0.225
agi_english/1798 seed24762 0.225
agi_english/274 seed24762 0.225
agi_english/2322 seed24762 0.225
agi_english/2088 seed24762 0.225
agi_english/2167 seed24762 0.225
agi_english/2086 seed24762 0.225
agi_english/1119 seed24762 0.225
agi_english/1641 seed24762 0.225
agi_english/1862 seed24762 0.225
agi_english/2502 seed24762 0.225
agi_english/945 seed24762 0.225
agi_english/956 seed24762 0.225
agi_english/873 seed24762 0.225
agi_english/915 seed24762 0.225
agi_english/466 seed24762 0.225
agi_english/910 seed24762 0.225
agi_english/1124 seed24762 0.225
agi_english/1864 seed24762 0.225
agi_english/2523 seed24762 0.225
agi_english/1827 seed24762 0.225
agi_english/1832 seed24762 0.225
agi_english/731 seed6529 0.213
agi_english/17 seed6529 0.213
agi_english/1664 seed6529 0.213
agi_english/1663 seed6529 0.213
agi_english/1652 seed6529 0.213
agi_english/2052 seed6529 0.213
agi_english/802 seed6529 0.213
agi_english/79 seed6529 0.213
agi_english/1779 seed6529 0.213
agi_english/1034 seed6529 0.213
agi_english/1007 seed6529 0.213
agi_english/691 seed6529 0.213
agi_english/1714 seed6529 0.213
agi_english/2448 seed6529 0.213
agi_english/1711 seed6529 0.213
agi_english/163 seed6529 0.213
agi_english/160 seed6529 0.213
agi_english/1882 seed6529 0.213
agi_english/1835 seed6529 0.213
agi_english/1811 seed6529 0.213
agi_english/343 seed6529 0.213
agi_english/325 seed6529 0.213
agi_english/157 seed6529 0.213
agi_english/494 seed6529 0.213
agi_english/1972 seed6529 0.213
agi_english/48 seed6529 0.213
agi_english/392 seed6529 0.213
agi_english/953 seed6529 0.213
agi_english/93 seed6529 0.213
agi_english/61 seed6529 0.213
agi_english/630 seed6529 0.213
agi_english/1127 seed6529 0.213
agi_english/995 seed6529 0.213

Suspect problems

These are 10 problems with the lowest correlation with the overall evaluation (i.e. better models tend to do worse on these. )

example_link pass1_of_ex tau
agi_english/2183 0.500 -0.745
agi_english/2430 0.600 -0.730
agi_english/1202 0.600 -0.730
agi_english/1448 0.600 -0.730
agi_english/2154 0.600 -0.730
agi_english/1543 0.600 -0.730
agi_english/2136 0.600 -0.730
agi_english/1081 0.300 -0.683
agi_english/570 0.500 -0.626
agi_english/1241 0.500 -0.626

Histogram of accuracies

Histogram of problems by the accuracy on each problem.

Histogram of difficulties

Histogram of problems by the minimum win rate to solve each problem.