There are 653 examples not solved by any model.
Solving some of these can be a good signal that your model is indeed better than leading models if these are good problems.
agi_english/1, agi_english/10, agi_english/1005, agi_english/1007, agi_english/1012, agi_english/1014, agi_english/1016, agi_english/1019, agi_english/1020, agi_english/1022, agi_english/1027, agi_english/1028, agi_english/103, agi_english/1034, agi_english/1039, agi_english/1043, agi_english/1046, agi_english/1049, agi_english/1054, agi_english/1057, agi_english/1061, agi_english/1062, agi_english/1064, agi_english/1065, agi_english/1069, agi_english/1071, agi_english/1072, agi_english/1076, agi_english/1078, agi_english/1081, agi_english/1083, agi_english/1085, agi_english/1086, agi_english/1088, agi_english/109, agi_english/1093, agi_english/1094, agi_english/1097, agi_english/1101, agi_english/1103, agi_english/1105, agi_english/1106, agi_english/1110, agi_english/1112, agi_english/1114, agi_english/1117, agi_english/1120, agi_english/1122, agi_english/1127, agi_english/1129, agi_english/1130, agi_english/1134, agi_english/1136, agi_english/1146, agi_english/1148, agi_english/1149, agi_english/116, agi_english/1160, agi_english/1161, agi_english/1163, agi_english/1165, agi_english/1169, agi_english/1177, agi_english/1178, agi_english/119, agi_english/1190, agi_english/1194, agi_english/1195, agi_english/120, agi_english/1204, agi_english/1206, agi_english/1207, agi_english/1208, agi_english/1209, agi_english/121, agi_english/1210, agi_english/1216, agi_english/1219, agi_english/1225, agi_english/1226, agi_english/1233, agi_english/1234, agi_english/1238, agi_english/1239, agi_english/124, agi_english/1244, agi_english/1245, agi_english/1247, agi_english/1254, agi_english/1255, agi_english/1258, agi_english/126, agi_english/1262, agi_english/1265, agi_english/1268, agi_english/1272, agi_english/1276, agi_english/1278, agi_english/1279, agi_english/128, agi_english/1283, agi_english/129, agi_english/1292, agi_english/1293, agi_english/1298, agi_english/1299, agi_english/1312, agi_english/1314, agi_english/1316, agi_english/1317, agi_english/132, agi_english/1321, agi_english/1323, agi_english/1327, agi_english/133, agi_english/1335, agi_english/1338, agi_english/1340, agi_english/1341, agi_english/1342, agi_english/1343, agi_english/1345, agi_english/1346, agi_english/1347, agi_english/1349, agi_english/135, agi_english/1351, agi_english/1352, agi_english/1357, agi_english/1360, agi_english/1362, agi_english/1363, agi_english/1367, agi_english/137, agi_english/1372, agi_english/1376, agi_english/1378, agi_english/1379, agi_english/1382, agi_english/1383, agi_english/1386, agi_english/1387, agi_english/139, agi_english/1396, agi_english/140, agi_english/1400, agi_english/1402, agi_english/1414, agi_english/1416, agi_english/1418, agi_english/1422, agi_english/1426, agi_english/143, agi_english/1431, agi_english/1441, agi_english/1442, agi_english/1443, agi_english/1445, agi_english/1447, agi_english/145, agi_english/1452, agi_english/146, agi_english/1460, agi_english/1461, agi_english/1462, agi_english/1466, agi_english/1468, agi_english/1473, agi_english/1481, agi_english/1482, agi_english/1483, agi_english/1491, agi_english/1492, agi_english/1493, agi_english/1498, agi_english/1499, agi_english/150, agi_english/1500, agi_english/1501, agi_english/1503, agi_english/1507, agi_english/1509, agi_english/1512, agi_english/1514, agi_english/1517, agi_english/1519, agi_english/1521, agi_english/1523, agi_english/1524, agi_english/1526, agi_english/1527, agi_english/153, agi_english/1531, agi_english/1532, agi_english/1538, agi_english/1540, agi_english/1541, agi_english/1544, agi_english/1547, agi_english/1548, agi_english/155, agi_english/1557, agi_english/156, agi_english/1560, agi_english/1565, agi_english/1566, agi_english/1570, agi_english/1572, agi_english/1573, agi_english/1578, agi_english/158, agi_english/1585, agi_english/1586, agi_english/1590, agi_english/1597, agi_english/1599, agi_english/160, agi_english/1605, agi_english/1609, agi_english/1610, agi_english/1614, agi_english/1618, agi_english/1625, agi_english/1627, agi_english/1628, agi_english/163, agi_english/1634, agi_english/1635, agi_english/1637, agi_english/1640, agi_english/1644, agi_english/1648, agi_english/1649, agi_english/1651, agi_english/1653, agi_english/1655, agi_english/1663, agi_english/1667, agi_english/1672, agi_english/1676, agi_english/1679, agi_english/1684, agi_english/1686, agi_english/1690, agi_english/1693, agi_english/1699, agi_english/17, agi_english/170, agi_english/1701, agi_english/1704, agi_english/1707, agi_english/1711, agi_english/1718, agi_english/1727, agi_english/1728, agi_english/1729, agi_english/1730, agi_english/1731, agi_english/1733, agi_english/1738, agi_english/174, agi_english/1740, agi_english/1741, agi_english/1742, agi_english/1743, agi_english/1745, agi_english/1746, agi_english/1748, agi_english/1751, agi_english/1752, agi_english/1753, agi_english/1754, agi_english/1757, agi_english/1758, agi_english/1759, agi_english/1765, agi_english/1766, agi_english/1767, agi_english/1768, agi_english/1769, agi_english/177, agi_english/1770, agi_english/1772, agi_english/1781, agi_english/1783, agi_english/1786, agi_english/1803, agi_english/1811, agi_english/1825, agi_english/1826, agi_english/183, agi_english/1838, agi_english/184, agi_english/1840, agi_english/1841, agi_english/1843, agi_english/1846, agi_english/185, agi_english/1850, agi_english/1856, agi_english/1857, agi_english/1858, agi_english/1865, agi_english/1866, agi_english/1871, agi_english/1874, agi_english/1876, agi_english/1877, agi_english/1880, agi_english/1881, agi_english/1883, agi_english/1886, agi_english/1889, agi_english/189, agi_english/1892, agi_english/1896, agi_english/1899, agi_english/190, agi_english/1901, agi_english/1904, agi_english/1906, agi_english/191, agi_english/1913, agi_english/1918, agi_english/192, agi_english/1928, agi_english/194, agi_english/1940, agi_english/1945, agi_english/2000, agi_english/2010, agi_english/2012, agi_english/2017, agi_english/2019, agi_english/2024, agi_english/2025, agi_english/2026, agi_english/2040, agi_english/2041, agi_english/2042, agi_english/2052, agi_english/2064, agi_english/2070, agi_english/2077, agi_english/208, agi_english/2083, agi_english/2084, agi_english/2092, agi_english/2093, agi_english/210, agi_english/2101, agi_english/2151, agi_english/2164, agi_english/2166, agi_english/2178, agi_english/218, agi_english/2185, agi_english/2187, agi_english/2199, agi_english/22, agi_english/2207, agi_english/2215, agi_english/2216, agi_english/222, agi_english/2225, agi_english/2229, agi_english/223, agi_english/225, agi_english/2268, agi_english/2292, agi_english/2296, agi_english/2299, agi_english/23, agi_english/2306, agi_english/2318, agi_english/2321, agi_english/2323, agi_english/2329, agi_english/2332, agi_english/2334, agi_english/2338, agi_english/2346, agi_english/2356, agi_english/2359, agi_english/2361, agi_english/2368, agi_english/2372, agi_english/2383, agi_english/239, agi_english/2390, agi_english/2396, agi_english/24, agi_english/2401, agi_english/2404, agi_english/2409, agi_english/2412, agi_english/2414, agi_english/2415, agi_english/2416, agi_english/2418, agi_english/2426, agi_english/2431, agi_english/2436, agi_english/2438, agi_english/2448, agi_english/245, agi_english/2452, agi_english/2456, agi_english/2458, agi_english/2459, agi_english/2462, agi_english/2466, agi_english/2468, agi_english/2469, agi_english/247, agi_english/2470, agi_english/248, agi_english/2483, agi_english/2489, agi_english/249, agi_english/2492, agi_english/2493, agi_english/2496, agi_english/2499, agi_english/250, agi_english/2501, agi_english/2505, agi_english/2514, agi_english/252, agi_english/2522, agi_english/2524, agi_english/2525, agi_english/2526, agi_english/2532, agi_english/2534, agi_english/2536, agi_english/2537, agi_english/2539, agi_english/2540, agi_english/256, agi_english/258, agi_english/263, agi_english/264, agi_english/267, agi_english/268, agi_english/269, agi_english/273, agi_english/274, agi_english/279, agi_english/280, agi_english/281, agi_english/283, agi_english/290, agi_english/291, agi_english/292, agi_english/294, agi_english/296, agi_english/298, agi_english/3, agi_english/301, agi_english/309, agi_english/313, agi_english/320, agi_english/327, agi_english/328, agi_english/33, agi_english/330, agi_english/335, agi_english/337, agi_english/34, agi_english/342, agi_english/343, agi_english/344, agi_english/358, agi_english/367, agi_english/374, agi_english/378, agi_english/381, agi_english/386, agi_english/388, agi_english/392, agi_english/396, agi_english/397, agi_english/402, agi_english/407, agi_english/408, agi_english/412, agi_english/413, agi_english/420, agi_english/423, agi_english/424, agi_english/434, agi_english/438, agi_english/44, agi_english/444, agi_english/446, agi_english/45, agi_english/452, agi_english/455, agi_english/459, agi_english/464, agi_english/471, agi_english/473, agi_english/48, agi_english/480, agi_english/488, agi_english/49, agi_english/490, agi_english/496, agi_english/497, agi_english/499, agi_english/501, agi_english/504, agi_english/508, agi_english/513, agi_english/515, agi_english/519, agi_english/52, agi_english/523, agi_english/532, agi_english/535, agi_english/538, agi_english/540, agi_english/541, agi_english/542, agi_english/548, agi_english/554, agi_english/555, agi_english/556, agi_english/557, agi_english/566, agi_english/568, agi_english/57, agi_english/571, agi_english/572, agi_english/576, agi_english/58, agi_english/590, agi_english/595, agi_english/60, agi_english/608, agi_english/609, agi_english/611, agi_english/613, agi_english/615, agi_english/617, agi_english/622, agi_english/626, agi_english/628, agi_english/632, agi_english/640, agi_english/642, agi_english/651, agi_english/652, agi_english/654, agi_english/657, agi_english/661, agi_english/692, agi_english/694, agi_english/696, agi_english/703, agi_english/706, agi_english/715, agi_english/716, agi_english/718, agi_english/719, agi_english/72, agi_english/720, agi_english/725, agi_english/73, agi_english/731, agi_english/732, agi_english/733, agi_english/738, agi_english/74, agi_english/740, agi_english/742, agi_english/746, agi_english/748, agi_english/753, agi_english/76, agi_english/760, agi_english/765, agi_english/766, agi_english/77, agi_english/771, agi_english/772, agi_english/773, agi_english/784, agi_english/79, agi_english/8, agi_english/80, agi_english/801, agi_english/803, agi_english/804, agi_english/807, agi_english/809, agi_english/811, agi_english/821, agi_english/829, agi_english/835, agi_english/845, agi_english/847, agi_english/849, agi_english/850, agi_english/853, agi_english/854, agi_english/859, agi_english/86, agi_english/861, agi_english/868, agi_english/869, agi_english/876, agi_english/881, agi_english/887, agi_english/891, agi_english/899, agi_english/90, agi_english/900, agi_english/901, agi_english/905, agi_english/907, agi_english/91, agi_english/910, agi_english/916, agi_english/920, agi_english/924, agi_english/926, agi_english/928, agi_english/935, agi_english/944, agi_english/948, agi_english/953, agi_english/954, agi_english/958, agi_english/962, agi_english/964, agi_english/966, agi_english/967, agi_english/968, agi_english/969, agi_english/97, agi_english/970, agi_english/973, agi_english/974, agi_english/977, agi_english/984, agi_english/986, agi_english/99, agi_english/995, agi_english/997, agi_english/998, agi_english/999
| example_link | model | min_pass1_of_model |
|---|---|---|
| agi_english/2086 | seed14309 | 0.243 |
| agi_english/2036 | seed14309 | 0.243 |
| agi_english/2206 | seed14309 | 0.243 |
| agi_english/2129 | seed14309 | 0.243 |
| agi_english/2120 | seed14309 | 0.243 |
| agi_english/2234 | seed14309 | 0.243 |
| agi_english/2239 | seed14309 | 0.243 |
| agi_english/2209 | seed14309 | 0.243 |
| agi_english/573 | seed14309 | 0.243 |
| agi_english/53 | seed14309 | 0.243 |
| agi_english/528 | seed14309 | 0.243 |
| agi_english/2256 | seed14309 | 0.243 |
| agi_english/2242 | seed14309 | 0.243 |
| agi_english/806 | seed14309 | 0.243 |
| agi_english/834 | seed14309 | 0.243 |
| agi_english/882 | seed14309 | 0.243 |
| agi_english/873 | seed14309 | 0.243 |
| agi_english/1955 | seed14309 | 0.243 |
| agi_english/579 | seed14309 | 0.243 |
| agi_english/570 | seed14309 | 0.243 |
| agi_english/547 | seed14309 | 0.243 |
| agi_english/583 | seed14309 | 0.243 |
| agi_english/371 | seed14309 | 0.243 |
| agi_english/393 | seed14309 | 0.243 |
| agi_english/1166 | seed14309 | 0.243 |
| agi_english/1198 | seed14309 | 0.243 |
| agi_english/690 | seed14309 | 0.243 |
| agi_english/670 | seed14309 | 0.243 |
| agi_english/667 | seed14309 | 0.243 |
| agi_english/481 | seed14309 | 0.243 |
| agi_english/476 | seed14309 | 0.243 |
| agi_english/470 | seed14309 | 0.243 |
| agi_english/1139 | seed14309 | 0.243 |
| agi_english/1291 | seed14309 | 0.243 |
| agi_english/1261 | seed14309 | 0.243 |
| agi_english/1288 | seed14309 | 0.243 |
| agi_english/1284 | seed14309 | 0.243 |
| agi_english/1807 | seed14309 | 0.243 |
| agi_english/1819 | seed14309 | 0.243 |
| agi_english/1666 | seed14309 | 0.243 |
| agi_english/1834 | seed14309 | 0.243 |
| agi_english/345 | seed14309 | 0.243 |
| agi_english/325 | seed14309 | 0.243 |
| agi_english/1579 | seed14309 | 0.243 |
| agi_english/1563 | seed14309 | 0.243 |
| agi_english/40 | seed14309 | 0.243 |
| agi_english/350 | seed14309 | 0.243 |
| agi_english/31 | seed14309 | 0.243 |
| agi_english/303 | seed14309 | 0.243 |
| agi_english/841 | seed14309 | 0.243 |
| agi_english/892 | seed14309 | 0.243 |
| agi_english/2324 | seed14309 | 0.243 |
| agi_english/848 | seed14309 | 0.243 |
| agi_english/790 | seed14309 | 0.243 |
| agi_english/793 | seed14309 | 0.243 |
| agi_english/823 | seed14309 | 0.243 |
| agi_english/785 | seed14309 | 0.243 |
| agi_english/1189 | seed14309 | 0.243 |
| agi_english/118 | seed14309 | 0.243 |
| agi_english/1221 | seed14309 | 0.243 |
| agi_english/722 | seed14309 | 0.243 |
| agi_english/2310 | seed14309 | 0.243 |
| agi_english/832 | seed14309 | 0.243 |
| agi_english/2144 | seed6115 | 0.242 |
| agi_english/2189 | seed6115 | 0.242 |
| agi_english/2062 | seed6115 | 0.242 |
| agi_english/2217 | seed6115 | 0.242 |
| agi_english/2020 | seed6115 | 0.242 |
| agi_english/1203 | seed6115 | 0.242 |
| agi_english/1425 | seed6115 | 0.242 |
| agi_english/1175 | seed6115 | 0.242 |
| agi_english/1167 | seed6115 | 0.242 |
| agi_english/1170 | seed6115 | 0.242 |
| agi_english/2249 | seed6115 | 0.242 |
| agi_english/2188 | seed6115 | 0.242 |
| agi_english/2196 | seed6115 | 0.242 |
| agi_english/2279 | seed6115 | 0.242 |
| agi_english/1623 | seed6115 | 0.242 |
| agi_english/1620 | seed6115 | 0.242 |
| agi_english/1716 | seed6115 | 0.242 |
| agi_english/1703 | seed6115 | 0.242 |
| agi_english/1691 | seed6115 | 0.242 |
| agi_english/1668 | seed6115 | 0.242 |
| agi_english/1898 | seed6115 | 0.242 |
| agi_english/1902 | seed6115 | 0.242 |
| agi_english/1862 | seed6115 | 0.242 |
| agi_english/1870 | seed6115 | 0.242 |
| agi_english/1879 | seed6115 | 0.242 |
| agi_english/1814 | seed6115 | 0.242 |
| agi_english/1311 | seed6115 | 0.242 |
| agi_english/1318 | seed6115 | 0.242 |
| agi_english/1281 | seed6115 | 0.242 |
| agi_english/1842 | seed6115 | 0.242 |
| agi_english/1496 | seed6115 | 0.242 |
| agi_english/425 | seed6115 | 0.242 |
| agi_english/1484 | seed6115 | 0.242 |
| agi_english/1475 | seed6115 | 0.242 |
| agi_english/1361 | seed6115 | 0.242 |
| agi_english/1365 | seed6115 | 0.242 |
| agi_english/1487 | seed6115 | 0.242 |
| agi_english/1421 | seed6115 | 0.242 |
| agi_english/1147 | seed6115 | 0.242 |
| agi_english/1339 | seed6115 | 0.242 |
| agi_english/1408 | seed6115 | 0.242 |
| agi_english/2541 | seed6115 | 0.242 |
| agi_english/1607 | seed6115 | 0.242 |
| agi_english/1567 | seed6115 | 0.242 |
| agi_english/2461 | seed6115 | 0.242 |
| agi_english/2531 | seed6115 | 0.242 |
| agi_english/1785 | seed6115 | 0.242 |
| agi_english/1779 | seed6115 | 0.242 |
| agi_english/2314 | seed6115 | 0.242 |
| agi_english/2293 | seed6115 | 0.242 |
| agi_english/1737 | seed6115 | 0.242 |
| agi_english/1675 | seed6115 | 0.242 |
| agi_english/1735 | seed6115 | 0.242 |
| agi_english/1722 | seed6115 | 0.242 |
| agi_english/2520 | seed6115 | 0.242 |
| agi_english/361 | seed6115 | 0.242 |
| agi_english/419 | seed6115 | 0.242 |
| agi_english/2465 | seed6115 | 0.242 |
| agi_english/2490 | seed6115 | 0.242 |
| agi_english/2479 | seed6115 | 0.242 |
| agi_english/2506 | seed6115 | 0.242 |
| agi_english/2518 | seed6115 | 0.242 |
| agi_english/1582 | seed6115 | 0.242 |
| agi_english/1601 | seed6115 | 0.242 |
| agi_english/940 | seed6115 | 0.242 |
| agi_english/943 | seed6115 | 0.242 |
| agi_english/982 | seed6115 | 0.242 |
| agi_english/2328 | seed6115 | 0.242 |
| agi_english/1537 | seed6115 | 0.242 |
| agi_english/2432 | seed6115 | 0.242 |
| agi_english/2376 | seed6115 | 0.242 |
| agi_english/2394 | seed6115 | 0.242 |
| agi_english/2457 | seed6115 | 0.242 |
| agi_english/2442 | seed6115 | 0.242 |
| agi_english/2422 | seed6115 | 0.242 |
| agi_english/1237 | seed6115 | 0.242 |
| agi_english/1197 | seed6115 | 0.242 |
| agi_english/1021 | seed6115 | 0.242 |
| agi_english/996 | seed149 | 0.241 |
| agi_english/994 | seed149 | 0.241 |
| agi_english/992 | seed149 | 0.241 |
| agi_english/1004 | seed149 | 0.241 |
| agi_english/1013 | seed149 | 0.241 |
| agi_english/1010 | seed149 | 0.241 |
| agi_english/704 | seed149 | 0.241 |
| agi_english/824 | seed149 | 0.241 |
| agi_english/938 | seed149 | 0.241 |
| agi_english/921 | seed149 | 0.241 |
| agi_english/1954 | seed149 | 0.241 |
| agi_english/929 | seed149 | 0.241 |
| agi_english/2013 | seed149 | 0.241 |
| agi_english/1123 | seed149 | 0.241 |
| agi_english/1108 | seed149 | 0.241 |
| agi_english/1107 | seed149 | 0.241 |
| agi_english/1080 | seed149 | 0.241 |
| agi_english/1528 | seed149 | 0.241 |
| agi_english/1533 | seed149 | 0.241 |
| agi_english/1549 | seed149 | 0.241 |
| agi_english/2307 | seed149 | 0.241 |
| agi_english/1763 | seed149 | 0.241 |
| agi_english/370 | seed149 | 0.241 |
| agi_english/2460 | seed149 | 0.241 |
| agi_english/2447 | seed149 | 0.241 |
| agi_english/2053 | seed149 | 0.241 |
| agi_english/631 | seed149 | 0.241 |
| agi_english/663 | seed149 | 0.241 |
| agi_english/1932 | seed149 | 0.241 |
| agi_english/1915 | seed149 | 0.241 |
| agi_english/1822 | seed149 | 0.241 |
| agi_english/1925 | seed149 | 0.241 |
| agi_english/641 | seed149 | 0.241 |
| agi_english/567 | seed149 | 0.241 |
| agi_english/591 | seed149 | 0.241 |
| agi_english/2255 | seed149 | 0.241 |
| agi_english/2131 | seed149 | 0.241 |
| agi_english/2121 | seed149 | 0.241 |
| agi_english/2142 | seed149 | 0.241 |
| agi_english/2260 | seed149 | 0.241 |
| agi_english/2014 | seed149 | 0.241 |
| agi_english/1067 | seed149 | 0.241 |
| agi_english/1042 | seed149 | 0.241 |
| agi_english/1074 | seed149 | 0.241 |
| agi_english/1073 | seed149 | 0.241 |
| agi_english/1089 | seed149 | 0.241 |
| agi_english/1100 | seed149 | 0.241 |
| agi_english/1910 | seed24762 | 0.234 |
| agi_english/1079 | seed24762 | 0.234 |
| agi_english/988 | seed24762 | 0.234 |
| agi_english/1131 | seed24762 | 0.234 |
| agi_english/1053 | seed24762 | 0.234 |
| agi_english/1063 | seed24762 | 0.234 |
| agi_english/1820 | seed24762 | 0.234 |
| agi_english/1070 | seed24762 | 0.234 |
| agi_english/1099 | seed24762 | 0.234 |
| agi_english/1051 | seed24762 | 0.234 |
| agi_english/989 | seed24762 | 0.234 |
| agi_english/942 | seed24762 | 0.234 |
| agi_english/360 | seed24762 | 0.234 |
| agi_english/956 | seed24762 | 0.234 |
| agi_english/993 | seed24762 | 0.234 |
| agi_english/1685 | seed30841 | 0.232 |
| agi_english/171 | seed30841 | 0.232 |
| agi_english/1641 | seed30841 | 0.232 |
| agi_english/1647 | seed30841 | 0.232 |
| agi_english/186 | seed30841 | 0.232 |
| agi_english/1812 | seed30841 | 0.232 |
| agi_english/1657 | seed30841 | 0.232 |
| agi_english/1916 | seed30841 | 0.232 |
| agi_english/219 | seed30841 | 0.232 |
| agi_english/2028 | seed30841 | 0.232 |
| agi_english/106 | seed30841 | 0.232 |
| agi_english/1996 | seed30841 | 0.232 |
| agi_english/1326 | seed30841 | 0.232 |
| agi_english/151 | seed30841 | 0.232 |
| agi_english/1790 | seed30841 | 0.232 |
| agi_english/1923 | seed30841 | 0.232 |
| agi_english/1894 | seed30841 | 0.232 |
| agi_english/1719 | seed30841 | 0.232 |
| agi_english/232 | seed30841 | 0.232 |
| agi_english/2545 | seed30841 | 0.232 |
| agi_english/1791 | seed30841 | 0.232 |
| agi_english/1799 | seed30841 | 0.232 |
| agi_english/1958 | seed30841 | 0.232 |
| agi_english/172 | seed30841 | 0.232 |
| agi_english/839 | seed17495 | 0.230 |
| agi_english/825 | seed17495 | 0.230 |
| agi_english/761 | seed17495 | 0.230 |
| agi_english/865 | seed17495 | 0.230 |
| agi_english/780 | seed17495 | 0.230 |
| agi_english/743 | seed17495 | 0.230 |
| agi_english/909 | seed17495 | 0.230 |
| agi_english/906 | seed17495 | 0.230 |
| agi_english/878 | seed17495 | 0.230 |
| agi_english/898 | seed17495 | 0.230 |
| agi_english/284 | seed17495 | 0.230 |
| agi_english/311 | seed17495 | 0.230 |
| agi_english/257 | seed17495 | 0.230 |
| agi_english/277 | seed17495 | 0.230 |
| agi_english/176 | seed17495 | 0.230 |
| agi_english/113 | seed17495 | 0.230 |
| agi_english/111 | seed17495 | 0.230 |
| agi_english/2445 | seed17495 | 0.230 |
| agi_english/108 | seed17495 | 0.230 |
| agi_english/93 | seed17495 | 0.230 |
| agi_english/98 | seed17495 | 0.230 |
| agi_english/983 | seed17495 | 0.230 |
| agi_english/457 | seed17495 | 0.230 |
| agi_english/1917 | seed17495 | 0.230 |
| agi_english/447 | seed17495 | 0.230 |
| agi_english/445 | seed17495 | 0.230 |
| agi_english/634 | seed17495 | 0.230 |
| agi_english/607 | seed17495 | 0.230 |
| agi_english/668 | seed17495 | 0.230 |
| agi_english/114 | seed17495 | 0.230 |
| agi_english/38 | seed17495 | 0.230 |
| agi_english/380 | seed17495 | 0.230 |
| agi_english/449 | seed17495 | 0.230 |
| agi_english/451 | seed17495 | 0.230 |
| agi_english/436 | seed17495 | 0.230 |
| agi_english/463 | seed17495 | 0.230 |
| agi_english/551 | seed17495 | 0.230 |
| agi_english/521 | seed17495 | 0.230 |
| agi_english/2124 | seed17495 | 0.230 |
| agi_english/498 | seed17495 | 0.230 |
| agi_english/1988 | seed17495 | 0.230 |
| agi_english/1056 | seed17495 | 0.230 |
| agi_english/1031 | seed27712 | 0.226 |
| agi_english/1033 | seed27712 | 0.226 |
| agi_english/1066 | seed27712 | 0.226 |
| agi_english/1068 | seed27712 | 0.226 |
| agi_english/1128 | seed27712 | 0.226 |
| agi_english/871 | seed27712 | 0.226 |
| agi_english/1126 | seed27712 | 0.226 |
| agi_english/925 | seed27712 | 0.226 |
| agi_english/955 | seed27712 | 0.226 |
| agi_english/951 | seed27712 | 0.226 |
| agi_english/950 | seed27712 | 0.226 |
| agi_english/1025 | seed27712 | 0.226 |
| agi_english/2317 | seed27712 | 0.226 |
| agi_english/2301 | seed27712 | 0.226 |
| agi_english/2274 | seed27712 | 0.226 |
| agi_english/2148 | seed27712 | 0.226 |
| agi_english/2410 | seed27712 | 0.226 |
| agi_english/562 | seed27712 | 0.226 |
| agi_english/644 | seed27712 | 0.226 |
| agi_english/676 | seed27712 | 0.226 |
| agi_english/558 | seed27712 | 0.226 |
| agi_english/516 | seed27712 | 0.226 |
| agi_english/1006 | seed27712 | 0.226 |
| agi_english/828 | seed27712 | 0.226 |
| agi_english/493 | seed27712 | 0.226 |
| agi_english/491 | seed27712 | 0.226 |
| agi_english/1520 | seed27712 | 0.226 |
| agi_english/347 | seed27712 | 0.226 |
| agi_english/265 | seed27712 | 0.226 |
| agi_english/336 | seed27712 | 0.226 |
| agi_english/422 | seed27712 | 0.226 |
| agi_english/2500 | seed27712 | 0.226 |
| agi_english/638 | seed27712 | 0.226 |
| agi_english/763 | seed27712 | 0.226 |
| agi_english/796 | seed27712 | 0.226 |
| agi_english/774 | seed27712 | 0.226 |
| agi_english/808 | seed27712 | 0.226 |
| agi_english/1002 | seed27712 | 0.226 |
| agi_english/695 | seed27712 | 0.226 |
| agi_english/699 | seed27712 | 0.226 |
| agi_english/752 | seed27712 | 0.226 |
| agi_english/2366 | seed16385 | 0.225 |
| agi_english/2393 | seed16385 | 0.225 |
| agi_english/237 | seed16385 | 0.225 |
| agi_english/240 | seed16385 | 0.225 |
| agi_english/1554 | seed16385 | 0.225 |
| agi_english/1552 | seed16385 | 0.225 |
| agi_english/2351 | seed16385 | 0.225 |
| agi_english/1550 | seed16385 | 0.225 |
| agi_english/221 | seed16385 | 0.225 |
| agi_english/2288 | seed16385 | 0.225 |
| agi_english/234 | seed16385 | 0.225 |
| agi_english/175 | seed16385 | 0.225 |
| agi_english/1571 | seed16385 | 0.225 |
| agi_english/205 | seed16385 | 0.225 |
| agi_english/1295 | seed16385 | 0.225 |
| agi_english/598 | seed16385 | 0.225 |
| agi_english/1411 | seed16385 | 0.225 |
| agi_english/1310 | seed16385 | 0.225 |
| agi_english/1319 | seed16385 | 0.225 |
| agi_english/1328 | seed16385 | 0.225 |
| agi_english/1348 | seed16385 | 0.225 |
| agi_english/141 | seed16385 | 0.225 |
| agi_english/1424 | seed16385 | 0.225 |
| agi_english/1587 | seed16385 | 0.225 |
| agi_english/161 | seed16385 | 0.225 |
| agi_english/1583 | seed16385 | 0.225 |
| agi_english/2474 | seed16385 | 0.225 |
| agi_english/28 | seed16385 | 0.225 |
| agi_english/30 | seed16385 | 0.225 |
| agi_english/272 | seed16385 | 0.225 |
| agi_english/2542 | seed16385 | 0.225 |
| agi_english/228 | seed16385 | 0.225 |
| agi_english/179 | seed16385 | 0.225 |
| agi_english/1639 | seed16385 | 0.225 |
| agi_english/1939 | seed16385 | 0.225 |
| agi_english/220 | seed16385 | 0.225 |
| agi_english/216 | seed16385 | 0.225 |
| agi_english/213 | seed16385 | 0.225 |
| agi_english/202 | seed16385 | 0.225 |
| agi_english/2360 | seed16385 | 0.225 |
| agi_english/200 | seed16385 | 0.225 |
| agi_english/1975 | seed16385 | 0.225 |
| agi_english/1979 | seed16385 | 0.225 |
| agi_english/1968 | seed16385 | 0.225 |
| agi_english/1974 | seed16385 | 0.225 |
| agi_english/165 | seed16385 | 0.225 |
| agi_english/1914 | seed16385 | 0.225 |
| agi_english/2003 | seed16385 | 0.225 |
| agi_english/2043 | seed16385 | 0.225 |
| agi_english/2073 | seed16385 | 0.225 |
| agi_english/196 | seed16385 | 0.225 |
| agi_english/89 | seed16385 | 0.225 |
| agi_english/88 | seed16385 | 0.225 |
| agi_english/107 | seed16385 | 0.225 |
| agi_english/105 | seed16385 | 0.225 |
| agi_english/87 | seed16385 | 0.225 |
| agi_english/1251 | seed16385 | 0.225 |
| agi_english/1227 | seed16385 | 0.225 |
| agi_english/1179 | seed16385 | 0.225 |
| agi_english/1205 | seed16385 | 0.225 |
| agi_english/67 | seed16385 | 0.225 |
| agi_english/1510 | seed16385 | 0.225 |
| agi_english/1480 | seed16385 | 0.225 |
| agi_english/1419 | seed16385 | 0.225 |
| agi_english/1285 | seed16385 | 0.225 |
| agi_english/1154 | seed16385 | 0.225 |
| agi_english/1158 | seed16385 | 0.225 |
| agi_english/1222 | seed16385 | 0.225 |
| agi_english/1214 | seed16385 | 0.225 |
| agi_english/62 | seed16385 | 0.225 |
| agi_english/1145 | seed16385 | 0.225 |
| agi_english/368 | seed16385 | 0.225 |
| agi_english/1464 | seed16385 | 0.225 |
| agi_english/1474 | seed16385 | 0.225 |
| agi_english/1471 | seed16385 | 0.225 |
| agi_english/1495 | seed16385 | 0.225 |
| agi_english/1264 | seed16385 | 0.225 |
| agi_english/32 | seed16385 | 0.225 |
| agi_english/4 | seed16385 | 0.225 |
| agi_english/439 | seed16385 | 0.225 |
| agi_english/1428 | seed16385 | 0.225 |
| agi_english/1430 | seed16385 | 0.225 |
| agi_english/142 | seed16385 | 0.225 |
| agi_english/2484 | seed16385 | 0.225 |
| agi_english/100 | seed24734 | 0.225 |
| agi_english/83 | seed24734 | 0.225 |
| agi_english/81 | seed24734 | 0.225 |
| agi_english/92 | seed24734 | 0.225 |
| agi_english/918 | seed24734 | 0.225 |
| agi_english/9 | seed24734 | 0.225 |
| agi_english/94 | seed24734 | 0.225 |
| agi_english/195 | seed24734 | 0.225 |
| agi_english/971 | seed24734 | 0.225 |
| agi_english/11 | seed24734 | 0.225 |
| agi_english/110 | seed24734 | 0.225 |
| agi_english/199 | seed24734 | 0.225 |
| agi_english/0 | seed24734 | 0.225 |
| agi_english/71 | seed24734 | 0.225 |
| agi_english/770 | seed24734 | 0.225 |
| agi_english/6 | seed24734 | 0.225 |
| agi_english/64 | seed24734 | 0.225 |
| agi_english/148 | seed24734 | 0.225 |
| agi_english/78 | seed24734 | 0.225 |
| agi_english/35 | seed24734 | 0.225 |
| agi_english/1829 | seed24734 | 0.225 |
| agi_english/1813 | seed24734 | 0.225 |
| agi_english/1815 | seed24734 | 0.225 |
| agi_english/2118 | seed24734 | 0.225 |
| agi_english/2061 | seed24734 | 0.225 |
| agi_english/212 | seed24734 | 0.225 |
| agi_english/215 | seed24734 | 0.225 |
| agi_english/18 | seed24734 | 0.225 |
| agi_english/167 | seed24734 | 0.225 |
| agi_english/235 | seed24734 | 0.225 |
| agi_english/1774 | seed24734 | 0.225 |
| agi_english/243 | seed24734 | 0.225 |
| agi_english/246 | seed24734 | 0.225 |
| agi_english/214 | seed24734 | 0.225 |
| agi_english/226 | seed24734 | 0.225 |
| agi_english/15 | seed24734 | 0.225 |
| agi_english/251 | seed24734 | 0.225 |
| agi_english/16 | seed24734 | 0.225 |
| agi_english/29 | seed24734 | 0.225 |
| agi_english/41 | seed24734 | 0.225 |
| agi_english/349 | seed24734 | 0.225 |
| agi_english/37 | seed24734 | 0.225 |
| agi_english/253 | seed24734 | 0.225 |
| agi_english/47 | seed24734 | 0.225 |
| agi_english/1576 | seed6529 | 0.209 |
| agi_english/2023 | seed6529 | 0.209 |
| agi_english/2089 | seed6529 | 0.209 |
| agi_english/1921 | seed6529 | 0.209 |
| agi_english/1873 | seed6529 | 0.209 |
| agi_english/1832 | seed6529 | 0.209 |
| agi_english/1942 | seed6529 | 0.209 |
| agi_english/1972 | seed6529 | 0.209 |
| agi_english/1574 | seed6529 | 0.209 |
| agi_english/1559 | seed6529 | 0.209 |
| agi_english/1535 | seed6529 | 0.209 |
| agi_english/1525 | seed6529 | 0.209 |
| agi_english/1788 | seed6529 | 0.209 |
| agi_english/1687 | seed6529 | 0.209 |
| agi_english/2339 | seed6529 | 0.209 |
| agi_english/1787 | seed6529 | 0.209 |
| agi_english/2127 | seed6529 | 0.209 |
| agi_english/1636 | seed6529 | 0.209 |
| agi_english/1632 | seed6529 | 0.209 |
| agi_english/1589 | seed6529 | 0.209 |
| agi_english/1633 | seed6529 | 0.209 |
| agi_english/1617 | seed6529 | 0.209 |
| agi_english/1630 | seed6529 | 0.209 |
| agi_english/1263 | seed6529 | 0.209 |
| agi_english/1260 | seed6529 | 0.209 |
| agi_english/1505 | seed6529 | 0.209 |
| agi_english/1511 | seed6529 | 0.209 |
| agi_english/1455 | seed6529 | 0.209 |
| agi_english/1488 | seed6529 | 0.209 |
| agi_english/1449 | seed6529 | 0.209 |
| agi_english/1450 | seed6529 | 0.209 |
| agi_english/1432 | seed6529 | 0.209 |
| agi_english/1412 | seed6529 | 0.209 |
| agi_english/1423 | seed6529 | 0.209 |
| agi_english/1427 | seed6529 | 0.209 |
| agi_english/1435 | seed6529 | 0.209 |
| agi_english/1616 | seed6529 | 0.209 |
| agi_english/1371 | seed6529 | 0.209 |
| agi_english/1405 | seed6529 | 0.209 |
| agi_english/1153 | seed6529 | 0.209 |
| agi_english/1152 | seed6529 | 0.209 |
| agi_english/1135 | seed6529 | 0.209 |
| agi_english/1368 | seed6529 | 0.209 |
| agi_english/1377 | seed6529 | 0.209 |
| agi_english/1394 | seed6529 | 0.209 |
| agi_english/1401 | seed6529 | 0.209 |
| agi_english/1315 | seed6529 | 0.209 |
| agi_english/1337 | seed6529 | 0.209 |
| agi_english/1329 | seed6529 | 0.209 |
| agi_english/1242 | seed6529 | 0.209 |
| agi_english/1217 | seed6529 | 0.209 |
| agi_english/1173 | seed6529 | 0.209 |
| agi_english/1159 | seed6529 | 0.209 |
| agi_english/1176 | seed6529 | 0.209 |
| agi_english/1200 | seed6529 | 0.209 |
These are 10 problems with the lowest correlation with the overall evaluation (i.e. better models tend to do worse on these. )
| example_link | pass1_of_ex | tau |
|---|---|---|
| agi_english/739 | 0.500 | -0.745 |
| agi_english/693 | 0.600 | -0.669 |
| agi_english/1023 | 0.500 | -0.626 |
| agi_english/1824 | 0.300 | -0.618 |
| agi_english/410 | 0.400 | -0.609 |
| agi_english/323 | 0.600 | -0.609 |
| agi_english/896 | 0.600 | -0.609 |
| agi_english/976 | 0.400 | -0.609 |
| agi_english/683 | 0.600 | -0.609 |
| agi_english/1935 | 0.700 | -0.553 |
Histogram of problems by the accuracy on each problem.
Histogram of problems by the minimum win rate to solve each problem.