There are 659 examples not solved by any model.
Solving some of these can be a good signal that your model is indeed better than leading models if these are good problems.
agi_english/1, agi_english/1000, agi_english/1004, agi_english/1010, agi_english/1012, agi_english/1013, agi_english/1019, agi_english/1020, agi_english/1021, agi_english/1027, agi_english/103, agi_english/1039, agi_english/1049, agi_english/1051, agi_english/1054, agi_english/1057, agi_english/1059, agi_english/1061, agi_english/1062, agi_english/1064, agi_english/1067, agi_english/1069, agi_english/107, agi_english/1076, agi_english/1078, agi_english/1080, agi_english/1083, agi_english/1085, agi_english/1087, agi_english/1088, agi_english/1089, agi_english/1093, agi_english/1094, agi_english/1099, agi_english/1100, agi_english/1101, agi_english/1105, agi_english/1107, agi_english/1108, agi_english/111, agi_english/1110, agi_english/1112, agi_english/1114, agi_english/1117, agi_english/1123, agi_english/1130, agi_english/1135, agi_english/1136, agi_english/1145, agi_english/1146, agi_english/1148, agi_english/1152, agi_english/1153, agi_english/1158, agi_english/1159, agi_english/1169, agi_english/1170, agi_english/1173, agi_english/1175, agi_english/1176, agi_english/1177, agi_english/1178, agi_english/1183, agi_english/1189, agi_english/119, agi_english/1190, agi_english/1194, agi_english/1195, agi_english/120, agi_english/1200, agi_english/1203, agi_english/1204, agi_english/1205, agi_english/1207, agi_english/1208, agi_english/1209, agi_english/1214, agi_english/1216, agi_english/1217, agi_english/1221, agi_english/1224, agi_english/1226, agi_english/1227, agi_english/1233, agi_english/1234, agi_english/1236, agi_english/1237, agi_english/1238, agi_english/124, agi_english/1242, agi_english/1243, agi_english/1244, agi_english/1247, agi_english/1251, agi_english/1257, agi_english/1258, agi_english/126, agi_english/1260, agi_english/1263, agi_english/1264, agi_english/1272, agi_english/1273, agi_english/1276, agi_english/1279, agi_english/128, agi_english/1281, agi_english/1284, agi_english/1285, agi_english/1287, agi_english/1288, agi_english/1291, agi_english/1292, agi_english/1293, agi_english/1298, agi_english/1299, agi_english/1310, agi_english/1311, agi_english/1312, agi_english/1315, agi_english/1316, agi_english/1318, agi_english/1326, agi_english/1327, agi_english/1329, agi_english/133, agi_english/1335, agi_english/1337, agi_english/1338, agi_english/1339, agi_english/1341, agi_english/1342, agi_english/1343, agi_english/1345, agi_english/1346, agi_english/1347, agi_english/1348, agi_english/1349, agi_english/135, agi_english/1352, agi_english/1357, agi_english/1359, agi_english/1360, agi_english/1361, agi_english/1362, agi_english/1365, agi_english/1368, agi_english/137, agi_english/1371, agi_english/1372, agi_english/1374, agi_english/1376, agi_english/1377, agi_english/1378, agi_english/1379, agi_english/1382, agi_english/1383, agi_english/1387, agi_english/1388, agi_english/1389, agi_english/139, agi_english/1394, agi_english/1396, agi_english/1397, agi_english/140, agi_english/1400, agi_english/1401, agi_english/1405, agi_english/1411, agi_english/1412, agi_english/1413, agi_english/1418, agi_english/1421, agi_english/1422, agi_english/1423, agi_english/1424, agi_english/1425, agi_english/1427, agi_english/1428, agi_english/143, agi_english/1431, agi_english/1432, agi_english/1435, agi_english/1441, agi_english/1442, agi_english/1444, agi_english/1445, agi_english/1447, agi_english/1449, agi_english/145, agi_english/1452, agi_english/1455, agi_english/1460, agi_english/1461, agi_english/1462, agi_english/1464, agi_english/1466, agi_english/1468, agi_english/1473, agi_english/1481, agi_english/1482, agi_english/1483, agi_english/1484, agi_english/1487, agi_english/1488, agi_english/149, agi_english/1495, agi_english/1496, agi_english/1498, agi_english/1499, agi_english/1500, agi_english/1503, agi_english/1505, agi_english/1507, agi_english/1509, agi_english/151, agi_english/1510, agi_english/1511, agi_english/1512, agi_english/1514, agi_english/1517, agi_english/1518, agi_english/1519, agi_english/1523, agi_english/1524, agi_english/1525, agi_english/1527, agi_english/153, agi_english/1532, agi_english/1535, agi_english/1537, agi_english/1538, agi_english/1540, agi_english/1541, agi_english/1542, agi_english/1544, agi_english/155, agi_english/1552, agi_english/1554, agi_english/1557, agi_english/1559, agi_english/156, agi_english/1560, agi_english/1561, agi_english/1563, agi_english/1565, agi_english/1566, agi_english/1567, agi_english/1571, agi_english/1572, agi_english/1573, agi_english/1574, agi_english/1576, agi_english/1578, agi_english/158, agi_english/1586, agi_english/1589, agi_english/1591, agi_english/1597, agi_english/1601, agi_english/1605, agi_english/1607, agi_english/1608, agi_english/1610, agi_english/1616, agi_english/1617, agi_english/1618, agi_english/1620, agi_english/1623, agi_english/1627, agi_english/1628, agi_english/1630, agi_english/1632, agi_english/1633, agi_english/1636, agi_english/1639, agi_english/1642, agi_english/1646, agi_english/1651, agi_english/1653, agi_english/1654, agi_english/1660, agi_english/1667, agi_english/1668, agi_english/1672, agi_english/1676, agi_english/1677, agi_english/1679, agi_english/1680, agi_english/1682, agi_english/1684, agi_english/1686, agi_english/1687, agi_english/1699, agi_english/170, agi_english/1700, agi_english/1701, agi_english/1703, agi_english/1706, agi_english/1707, agi_english/171, agi_english/1716, agi_english/172, agi_english/1721, agi_english/1722, agi_english/1727, agi_english/1731, agi_english/1733, agi_english/1737, agi_english/1740, agi_english/1741, agi_english/1742, agi_english/1743, agi_english/1745, agi_english/1746, agi_english/1748, agi_english/1751, agi_english/1752, agi_english/1753, agi_english/1754, agi_english/1757, agi_english/1758, agi_english/1759, agi_english/1765, agi_english/1766, agi_english/1767, agi_english/1768, agi_english/1769, agi_english/177, agi_english/1770, agi_english/1772, agi_english/1778, agi_english/1781, agi_english/1782, agi_english/1783, agi_english/1785, agi_english/1788, agi_english/1789, agi_english/1794, agi_english/1796, agi_english/1799, agi_english/1800, agi_english/1802, agi_english/1804, agi_english/1810, agi_english/1813, agi_english/1814, agi_english/1815, agi_english/1816, agi_english/1822, agi_english/1824, agi_english/1825, agi_english/1830, agi_english/1834, agi_english/184, agi_english/1840, agi_english/1841, agi_english/1843, agi_english/1846, agi_english/1847, agi_english/1848, agi_english/185, agi_english/1850, agi_english/1856, agi_english/1858, agi_english/1860, agi_english/1866, agi_english/1868, agi_english/1871, agi_english/1873, agi_english/1874, agi_english/1878, agi_english/1881, agi_english/1889, agi_english/189, agi_english/1891, agi_english/1892, agi_english/1893, agi_english/1898, agi_english/1899, agi_english/190, agi_english/1901, agi_english/1902, agi_english/1903, agi_english/1906, agi_english/191, agi_english/1917, agi_english/1918, agi_english/1933, agi_english/1935, agi_english/1936, agi_english/1937, agi_english/1942, agi_english/1943, agi_english/1961, agi_english/1964, agi_english/1981, agi_english/1982, agi_english/1983, agi_english/1991, agi_english/1992, agi_english/2002, agi_english/2007, agi_english/2012, agi_english/2017, agi_english/2023, agi_english/2024, agi_english/2025, agi_english/2040, agi_english/2041, agi_english/2042, agi_english/2044, agi_english/2061, agi_english/2069, agi_english/2077, agi_english/208, agi_english/2080, agi_english/2081, agi_english/2083, agi_english/210, agi_english/2101, agi_english/2104, agi_english/2110, agi_english/2113, agi_english/2116, agi_english/2117, agi_english/2123, agi_english/2124, agi_english/2125, agi_english/2126, agi_english/2139, agi_english/2141, agi_english/2142, agi_english/2144, agi_english/2148, agi_english/2149, agi_english/2156, agi_english/2166, agi_english/2170, agi_english/2178, agi_english/218, agi_english/2187, agi_english/2188, agi_english/2189, agi_english/219, agi_english/2191, agi_english/2196, agi_english/2198, agi_english/22, agi_english/2208, agi_english/2210, agi_english/2213, agi_english/2214, agi_english/2217, agi_english/222, agi_english/2223, agi_english/2225, agi_english/2227, agi_english/2229, agi_english/223, agi_english/2230, agi_english/2235, agi_english/2236, agi_english/2241, agi_english/2243, agi_english/2246, agi_english/2249, agi_english/2251, agi_english/2255, agi_english/2262, agi_english/2266, agi_english/2274, agi_english/2275, agi_english/2279, agi_english/2280, agi_english/2287, agi_english/2293, agi_english/2294, agi_english/2296, agi_english/2297, agi_english/2299, agi_english/230, agi_english/2301, agi_english/2304, agi_english/2306, agi_english/2314, agi_english/2317, agi_english/232, agi_english/2320, agi_english/2321, agi_english/2323, agi_english/2328, agi_english/2330, agi_english/2333, agi_english/2340, agi_english/2341, agi_english/2342, agi_english/2348, agi_english/2349, agi_english/2353, agi_english/2355, agi_english/2357, agi_english/2359, agi_english/2360, agi_english/2365, agi_english/2369, agi_english/237, agi_english/2370, agi_english/2373, agi_english/2376, agi_english/2378, agi_english/2380, agi_english/2381, agi_english/2390, agi_english/2394, agi_english/2395, agi_english/2396, agi_english/24, agi_english/2401, agi_english/2405, agi_english/2408, agi_english/2409, agi_english/2418, agi_english/2421, agi_english/2422, agi_english/2427, agi_english/2432, agi_english/2436, agi_english/2442, agi_english/2445, agi_english/2449, agi_english/245, agi_english/2450, agi_english/2452, agi_english/2457, agi_english/2460, agi_english/2461, agi_english/2465, agi_english/247, agi_english/2478, agi_english/2479, agi_english/248, agi_english/2485, agi_english/2488, agi_english/249, agi_english/2490, agi_english/2497, agi_english/250, agi_english/2504, agi_english/2505, agi_english/2506, agi_english/2507, agi_english/2511, agi_english/2513, agi_english/2518, agi_english/2519, agi_english/2520, agi_english/2521, agi_english/2533, agi_english/2534, agi_english/2535, agi_english/2541, agi_english/2542, agi_english/2543, agi_english/2545, agi_english/273, agi_english/283, agi_english/298, agi_english/311, agi_english/313, agi_english/320, agi_english/33, agi_english/344, agi_english/347, agi_english/349, agi_english/365, agi_english/367, agi_english/369, agi_english/38, agi_english/405, agi_english/424, agi_english/431, agi_english/435, agi_english/445, agi_english/45, agi_english/453, agi_english/473, agi_english/488, agi_english/498, agi_english/499, agi_english/504, agi_english/521, agi_english/532, agi_english/535, agi_english/540, agi_english/552, agi_english/556, agi_english/576, agi_english/58, agi_english/580, agi_english/586, agi_english/60, agi_english/613, agi_english/644, agi_english/648, agi_english/650, agi_english/670, agi_english/680, agi_english/692, agi_english/699, agi_english/718, agi_english/73, agi_english/732, agi_english/74, agi_english/740, agi_english/742, agi_english/76, agi_english/766, agi_english/77, agi_english/778, agi_english/798, agi_english/8, agi_english/800, agi_english/811, agi_english/817, agi_english/853, agi_english/855, agi_english/86, agi_english/901, agi_english/905, agi_english/908, agi_english/911, agi_english/919, agi_english/920, agi_english/921, agi_english/929, agi_english/936, agi_english/938, agi_english/939, agi_english/940, agi_english/943, agi_english/948, agi_english/954, agi_english/958, agi_english/962, agi_english/967, agi_english/97, agi_english/970, agi_english/972, agi_english/974, agi_english/977, agi_english/979, agi_english/982, agi_english/986, agi_english/988, agi_english/99, agi_english/994, agi_english/996, agi_english/997, agi_english/998
| example_link | model | min_pass1_of_model |
|---|---|---|
| agi_english/753 | seed17495 | 0.247 |
| agi_english/761 | seed17495 | 0.247 |
| agi_english/780 | seed17495 | 0.247 |
| agi_english/725 | seed17495 | 0.247 |
| agi_english/696 | seed17495 | 0.247 |
| agi_english/748 | seed17495 | 0.247 |
| agi_english/676 | seed17495 | 0.247 |
| agi_english/652 | seed17495 | 0.247 |
| agi_english/651 | seed17495 | 0.247 |
| agi_english/678 | seed17495 | 0.247 |
| agi_english/605 | seed17495 | 0.247 |
| agi_english/463 | seed17495 | 0.247 |
| agi_english/46 | seed17495 | 0.247 |
| agi_english/436 | seed17495 | 0.247 |
| agi_english/513 | seed17495 | 0.247 |
| agi_english/501 | seed17495 | 0.247 |
| agi_english/428 | seed17495 | 0.247 |
| agi_english/1450 | seed17495 | 0.247 |
| agi_english/1443 | seed17495 | 0.247 |
| agi_english/332 | seed17495 | 0.247 |
| agi_english/261 | seed17495 | 0.247 |
| agi_english/269 | seed17495 | 0.247 |
| agi_english/327 | seed17495 | 0.247 |
| agi_english/393 | seed17495 | 0.247 |
| agi_english/383 | seed17495 | 0.247 |
| agi_english/434 | seed17495 | 0.247 |
| agi_english/2463 | seed17495 | 0.247 |
| agi_english/2453 | seed17495 | 0.247 |
| agi_english/2435 | seed17495 | 0.247 |
| agi_english/2539 | seed17495 | 0.247 |
| agi_english/186 | seed17495 | 0.247 |
| agi_english/1959 | seed17495 | 0.247 |
| agi_english/2143 | seed17495 | 0.247 |
| agi_english/2165 | seed17495 | 0.247 |
| agi_english/1775 | seed17495 | 0.247 |
| agi_english/2082 | seed17495 | 0.247 |
| agi_english/1717 | seed17495 | 0.247 |
| agi_english/1985 | seed17495 | 0.247 |
| agi_english/2064 | seed17495 | 0.247 |
| agi_english/1669 | seed17495 | 0.247 |
| agi_english/202 | seed17495 | 0.247 |
| agi_english/1988 | seed17495 | 0.247 |
| agi_english/1966 | seed17495 | 0.247 |
| agi_english/849 | seed17495 | 0.247 |
| agi_english/1926 | seed17495 | 0.247 |
| agi_english/2286 | seed17495 | 0.247 |
| agi_english/2250 | seed17495 | 0.247 |
| agi_english/2194 | seed17495 | 0.247 |
| agi_english/2446 | seed17495 | 0.247 |
| agi_english/2337 | seed17495 | 0.247 |
| agi_english/239 | seed17495 | 0.247 |
| agi_english/2391 | seed17495 | 0.247 |
| agi_english/2411 | seed17495 | 0.247 |
| agi_english/2413 | seed17495 | 0.247 |
| agi_english/2309 | seed17495 | 0.247 |
| agi_english/2272 | seed17495 | 0.247 |
| agi_english/2197 | seed17495 | 0.247 |
| agi_english/2433 | seed17495 | 0.247 |
| agi_english/291 | seed17495 | 0.247 |
| agi_english/281 | seed17495 | 0.247 |
| agi_english/72 | seed16385 | 0.243 |
| agi_english/1262 | seed16385 | 0.243 |
| agi_english/1154 | seed16385 | 0.243 |
| agi_english/1265 | seed16385 | 0.243 |
| agi_english/28 | seed16385 | 0.243 |
| agi_english/321 | seed16385 | 0.243 |
| agi_english/342 | seed16385 | 0.243 |
| agi_english/32 | seed16385 | 0.243 |
| agi_english/34 | seed16385 | 0.243 |
| agi_english/1319 | seed16385 | 0.243 |
| agi_english/132 | seed16385 | 0.243 |
| agi_english/1351 | seed16385 | 0.243 |
| agi_english/146 | seed16385 | 0.243 |
| agi_english/141 | seed16385 | 0.243 |
| agi_english/44 | seed16385 | 0.243 |
| agi_english/441 | seed16385 | 0.243 |
| agi_english/4 | seed16385 | 0.243 |
| agi_english/1474 | seed16385 | 0.243 |
| agi_english/1493 | seed16385 | 0.243 |
| agi_english/142 | seed16385 | 0.243 |
| agi_english/411 | seed16385 | 0.243 |
| agi_english/1261 | seed16385 | 0.243 |
| agi_english/1254 | seed16385 | 0.243 |
| agi_english/1225 | seed16385 | 0.243 |
| agi_english/1304 | seed16385 | 0.243 |
| agi_english/656 | seed16385 | 0.243 |
| agi_english/67 | seed16385 | 0.243 |
| agi_english/1163 | seed16385 | 0.243 |
| agi_english/640 | seed16385 | 0.243 |
| agi_english/62 | seed16385 | 0.243 |
| agi_english/585 | seed16385 | 0.243 |
| agi_english/607 | seed16385 | 0.243 |
| agi_english/52 | seed16385 | 0.243 |
| agi_english/558 | seed16385 | 0.243 |
| agi_english/568 | seed16385 | 0.243 |
| agi_english/49 | seed16385 | 0.243 |
| agi_english/746 | seed16385 | 0.243 |
| agi_english/80 | seed16385 | 0.243 |
| agi_english/220 | seed16385 | 0.243 |
| agi_english/2415 | seed16385 | 0.243 |
| agi_english/1546 | seed16385 | 0.243 |
| agi_english/2231 | seed16385 | 0.243 |
| agi_english/225 | seed16385 | 0.243 |
| agi_english/252 | seed16385 | 0.243 |
| agi_english/1625 | seed16385 | 0.243 |
| agi_english/161 | seed16385 | 0.243 |
| agi_english/165 | seed16385 | 0.243 |
| agi_english/1635 | seed16385 | 0.243 |
| agi_english/1570 | seed16385 | 0.243 |
| agi_english/150 | seed16385 | 0.243 |
| agi_english/1583 | seed16385 | 0.243 |
| agi_english/221 | seed16385 | 0.243 |
| agi_english/205 | seed16385 | 0.243 |
| agi_english/2011 | seed16385 | 0.243 |
| agi_english/200 | seed16385 | 0.243 |
| agi_english/2078 | seed16385 | 0.243 |
| agi_english/216 | seed16385 | 0.243 |
| agi_english/196 | seed16385 | 0.243 |
| agi_english/192 | seed16385 | 0.243 |
| agi_english/194 | seed16385 | 0.243 |
| agi_english/1859 | seed16385 | 0.243 |
| agi_english/1806 | seed16385 | 0.243 |
| agi_english/89 | seed16385 | 0.243 |
| agi_english/88 | seed16385 | 0.243 |
| agi_english/898 | seed16385 | 0.243 |
| agi_english/87 | seed16385 | 0.243 |
| agi_english/109 | seed16385 | 0.243 |
| agi_english/1047 | seed16385 | 0.243 |
| agi_english/679 | seed149 | 0.241 |
| agi_english/1529 | seed149 | 0.241 |
| agi_english/597 | seed149 | 0.241 |
| agi_english/658 | seed149 | 0.241 |
| agi_english/567 | seed149 | 0.241 |
| agi_english/587 | seed149 | 0.241 |
| agi_english/526 | seed149 | 0.241 |
| agi_english/455 | seed149 | 0.241 |
| agi_english/833 | seed149 | 0.241 |
| agi_english/300 | seed149 | 0.241 |
| agi_english/2522 | seed149 | 0.241 |
| agi_english/1547 | seed149 | 0.241 |
| agi_english/1851 | seed149 | 0.241 |
| agi_english/1829 | seed149 | 0.241 |
| agi_english/357 | seed149 | 0.241 |
| agi_english/449 | seed149 | 0.241 |
| agi_english/429 | seed149 | 0.241 |
| agi_english/2073 | seed149 | 0.241 |
| agi_english/747 | seed149 | 0.241 |
| agi_english/2339 | seed149 | 0.241 |
| agi_english/2318 | seed149 | 0.241 |
| agi_english/2043 | seed149 | 0.241 |
| agi_english/2290 | seed149 | 0.241 |
| agi_english/1229 | seed149 | 0.241 |
| agi_english/707 | seed149 | 0.241 |
| agi_english/1693 | seed149 | 0.241 |
| agi_english/1710 | seed149 | 0.241 |
| agi_english/2129 | seed27712 | 0.237 |
| agi_english/422 | seed27712 | 0.237 |
| agi_english/260 | seed27712 | 0.237 |
| agi_english/318 | seed27712 | 0.237 |
| agi_english/1879 | seed27712 | 0.237 |
| agi_english/1969 | seed27712 | 0.237 |
| agi_english/1634 | seed27712 | 0.237 |
| agi_english/284 | seed27712 | 0.237 |
| agi_english/2070 | seed27712 | 0.237 |
| agi_english/1675 | seed27712 | 0.237 |
| agi_english/743 | seed27712 | 0.237 |
| agi_english/785 | seed27712 | 0.237 |
| agi_english/1736 | seed27712 | 0.237 |
| agi_english/2114 | seed27712 | 0.237 |
| agi_english/713 | seed27712 | 0.237 |
| agi_english/1071 | seed27712 | 0.237 |
| agi_english/1222 | seed27712 | 0.237 |
| agi_english/2087 | seed27712 | 0.237 |
| agi_english/2253 | seed27712 | 0.237 |
| agi_english/2218 | seed27712 | 0.237 |
| agi_english/2206 | seed27712 | 0.237 |
| agi_english/2090 | seed27712 | 0.237 |
| agi_english/1720 | seed27712 | 0.237 |
| agi_english/1520 | seed27712 | 0.237 |
| agi_english/2336 | seed27712 | 0.237 |
| agi_english/2175 | seed27712 | 0.237 |
| agi_english/2325 | seed27712 | 0.237 |
| agi_english/2310 | seed27712 | 0.237 |
| agi_english/1323 | seed27712 | 0.237 |
| agi_english/326 | seed27712 | 0.237 |
| agi_english/562 | seed27712 | 0.237 |
| agi_english/457 | seed27712 | 0.237 |
| agi_english/491 | seed27712 | 0.237 |
| agi_english/406 | seed27712 | 0.237 |
| agi_english/1792 | seed27712 | 0.237 |
| agi_english/59 | seed27712 | 0.237 |
| agi_english/611 | seed27712 | 0.237 |
| agi_english/1133 | seed27712 | 0.237 |
| agi_english/1728 | seed27712 | 0.237 |
| agi_english/714 | seed27712 | 0.237 |
| agi_english/1295 | seed27712 | 0.237 |
| agi_english/2254 | seed27712 | 0.237 |
| agi_english/1016 | seed14309 | 0.237 |
| agi_english/2112 | seed14309 | 0.237 |
| agi_english/751 | seed14309 | 0.237 |
| agi_english/2420 | seed14309 | 0.237 |
| agi_english/1685 | seed14309 | 0.237 |
| agi_english/1729 | seed14309 | 0.237 |
| agi_english/2481 | seed14309 | 0.237 |
| agi_english/2059 | seed14309 | 0.237 |
| agi_english/1666 | seed14309 | 0.237 |
| agi_english/1657 | seed14309 | 0.237 |
| agi_english/2008 | seed14309 | 0.237 |
| agi_english/1723 | seed14309 | 0.237 |
| agi_english/812 | seed14309 | 0.237 |
| agi_english/426 | seed14309 | 0.237 |
| agi_english/895 | seed14309 | 0.237 |
| agi_english/373 | seed14309 | 0.237 |
| agi_english/563 | seed14309 | 0.237 |
| agi_english/1780 | seed14309 | 0.237 |
| agi_english/583 | seed14309 | 0.237 |
| agi_english/903 | seed14309 | 0.237 |
| agi_english/2528 | seed14309 | 0.237 |
| agi_english/1807 | seed14309 | 0.237 |
| agi_english/1890 | seed14309 | 0.237 |
| agi_english/1883 | seed14309 | 0.237 |
| agi_english/370 | seed14309 | 0.237 |
| agi_english/293 | seed14309 | 0.237 |
| agi_english/698 | seed14309 | 0.237 |
| agi_english/2119 | seed14309 | 0.237 |
| agi_english/2084 | seed14309 | 0.237 |
| agi_english/1219 | seed24734 | 0.236 |
| agi_english/296 | seed24734 | 0.236 |
| agi_english/337 | seed24734 | 0.236 |
| agi_english/1967 | seed24734 | 0.236 |
| agi_english/1865 | seed24734 | 0.236 |
| agi_english/1857 | seed24734 | 0.236 |
| agi_english/1838 | seed24734 | 0.236 |
| agi_english/371 | seed24734 | 0.236 |
| agi_english/1896 | seed24734 | 0.236 |
| agi_english/657 | seed24734 | 0.236 |
| agi_english/528 | seed24734 | 0.236 |
| agi_english/572 | seed24734 | 0.236 |
| agi_english/980 | seed24734 | 0.236 |
| agi_english/841 | seed24734 | 0.236 |
| agi_english/476 | seed24734 | 0.236 |
| agi_english/617 | seed24734 | 0.236 |
| agi_english/1340 | seed24734 | 0.236 |
| agi_english/309 | seed24734 | 0.236 |
| agi_english/2473 | seed24734 | 0.236 |
| agi_english/770 | seed24734 | 0.236 |
| agi_english/779 | seed24734 | 0.236 |
| agi_english/738 | seed24734 | 0.236 |
| agi_english/806 | seed24734 | 0.236 |
| agi_english/1116 | seed24734 | 0.236 |
| agi_english/715 | seed24734 | 0.236 |
| agi_english/1092 | seed24734 | 0.236 |
| agi_english/829 | seed24734 | 0.236 |
| agi_english/1038 | seed24734 | 0.236 |
| agi_english/1055 | seed24734 | 0.236 |
| agi_english/1001 | seed24734 | 0.236 |
| agi_english/1017 | seed24734 | 0.236 |
| agi_english/13 | seed6115 | 0.233 |
| agi_english/775 | seed6115 | 0.233 |
| agi_english/831 | seed6115 | 0.233 |
| agi_english/2054 | seed6115 | 0.233 |
| agi_english/1777 | seed6115 | 0.233 |
| agi_english/816 | seed6115 | 0.233 |
| agi_english/805 | seed6115 | 0.233 |
| agi_english/791 | seed6115 | 0.233 |
| agi_english/2062 | seed6115 | 0.233 |
| agi_english/1950 | seed6115 | 0.233 |
| agi_english/9 | seed6115 | 0.233 |
| agi_english/3 | seed6115 | 0.233 |
| agi_english/1821 | seed6115 | 0.233 |
| agi_english/1363 | seed6115 | 0.233 |
| agi_english/1386 | seed6115 | 0.233 |
| agi_english/1812 | seed6115 | 0.233 |
| agi_english/1912 | seed6115 | 0.233 |
| agi_english/90 | seed6115 | 0.233 |
| agi_english/1801 | seed6115 | 0.233 |
| agi_english/1747 | seed30841 | 0.231 |
| agi_english/1691 | seed30841 | 0.231 |
| agi_english/1696 | seed30841 | 0.231 |
| agi_english/1690 | seed30841 | 0.231 |
| agi_english/1704 | seed30841 | 0.231 |
| agi_english/824 | seed30841 | 0.231 |
| agi_english/1314 | seed30841 | 0.231 |
| agi_english/1283 | seed30841 | 0.231 |
| agi_english/1005 | seed30841 | 0.231 |
| agi_english/1065 | seed30841 | 0.231 |
| agi_english/1043 | seed30841 | 0.231 |
| agi_english/1073 | seed30841 | 0.231 |
| agi_english/1072 | seed30841 | 0.231 |
| agi_english/2456 | seed30841 | 0.231 |
| agi_english/2467 | seed30841 | 0.231 |
| agi_english/2439 | seed30841 | 0.231 |
| agi_english/243 | seed30841 | 0.231 |
| agi_english/2382 | seed30841 | 0.231 |
| agi_english/1356 | seed30841 | 0.231 |
| agi_english/2530 | seed30841 | 0.231 |
| agi_english/1649 | seed30841 | 0.231 |
| agi_english/1599 | seed30841 | 0.231 |
| agi_english/1590 | seed30841 | 0.231 |
| agi_english/1585 | seed30841 | 0.231 |
| agi_english/1587 | seed30841 | 0.231 |
| agi_english/1640 | seed30841 | 0.231 |
| agi_english/1644 | seed30841 | 0.231 |
| agi_english/1637 | seed30841 | 0.231 |
| agi_english/1648 | seed30841 | 0.231 |
| agi_english/1718 | seed30841 | 0.231 |
| agi_english/2417 | seed30841 | 0.231 |
| agi_english/1712 | seed30841 | 0.231 |
| agi_english/1528 | seed30841 | 0.231 |
| agi_english/1526 | seed30841 | 0.231 |
| agi_english/2476 | seed30841 | 0.231 |
| agi_english/1655 | seed30841 | 0.231 |
| agi_english/2284 | seed30841 | 0.231 |
| agi_english/1735 | seed30841 | 0.231 |
| agi_english/1730 | seed30841 | 0.231 |
| agi_english/1278 | seed30841 | 0.231 |
| agi_english/1255 | seed30841 | 0.231 |
| agi_english/1245 | seed30841 | 0.231 |
| agi_english/1853 | seed30841 | 0.231 |
| agi_english/1870 | seed30841 | 0.231 |
| agi_english/1886 | seed30841 | 0.231 |
| agi_english/1122 | seed30841 | 0.231 |
| agi_english/621 | seed30841 | 0.231 |
| agi_english/1161 | seed30841 | 0.231 |
| agi_english/1165 | seed30841 | 0.231 |
| agi_english/425 | seed30841 | 0.231 |
| agi_english/926 | seed30841 | 0.231 |
| agi_english/1149 | seed30841 | 0.231 |
| agi_english/1141 | seed30841 | 0.231 |
| agi_english/1206 | seed30841 | 0.231 |
| agi_english/1197 | seed30841 | 0.231 |
| agi_english/1139 | seed30841 | 0.231 |
| agi_english/1167 | seed30841 | 0.231 |
| agi_english/1791 | seed30841 | 0.231 |
| agi_english/992 | seed30841 | 0.231 |
| agi_english/964 | seed30841 | 0.231 |
| agi_english/1426 | seed30841 | 0.231 |
| agi_english/1494 | seed30841 | 0.231 |
| agi_english/1416 | seed30841 | 0.231 |
| agi_english/1475 | seed30841 | 0.231 |
| agi_english/1480 | seed30841 | 0.231 |
| agi_english/1438 | seed30841 | 0.231 |
| agi_english/1430 | seed30841 | 0.231 |
| agi_english/1429 | seed30841 | 0.231 |
| agi_english/1467 | seed30841 | 0.231 |
| agi_english/1414 | seed30841 | 0.231 |
| agi_english/1913 | seed30841 | 0.231 |
| agi_english/1408 | seed30841 | 0.231 |
| agi_english/1522 | seed30841 | 0.231 |
| agi_english/1582 | seed30841 | 0.231 |
| agi_english/1581 | seed30841 | 0.231 |
| agi_english/1579 | seed30841 | 0.231 |
| agi_english/1531 | seed30841 | 0.231 |
| agi_english/1919 | seed30841 | 0.231 |
| agi_english/1367 | seed30841 | 0.231 |
| agi_english/1402 | seed30841 | 0.231 |
| agi_english/1852 | seed30841 | 0.231 |
| agi_english/1867 | seed30841 | 0.231 |
| agi_english/1842 | seed30841 | 0.231 |
| agi_english/2524 | seed30841 | 0.231 |
| agi_english/2503 | seed30841 | 0.231 |
| agi_english/1395 | seed30841 | 0.231 |
| agi_english/1492 | seed30841 | 0.231 |
| agi_english/1491 | seed30841 | 0.231 |
| agi_english/1328 | seed30841 | 0.231 |
| agi_english/1849 | seed30841 | 0.231 |
| agi_english/913 | seed30841 | 0.231 |
| agi_english/1040 | seed24762 | 0.225 |
| agi_english/1037 | seed24762 | 0.225 |
| agi_english/1014 | seed24762 | 0.225 |
| agi_english/1121 | seed24762 | 0.225 |
| agi_english/1070 | seed24762 | 0.225 |
| agi_english/1053 | seed24762 | 0.225 |
| agi_english/1052 | seed24762 | 0.225 |
| agi_english/1041 | seed24762 | 0.225 |
| agi_english/1058 | seed24762 | 0.225 |
| agi_english/1063 | seed24762 | 0.225 |
| agi_english/1096 | seed24762 | 0.225 |
| agi_english/1091 | seed24762 | 0.225 |
| agi_english/1111 | seed24762 | 0.225 |
| agi_english/1115 | seed24762 | 0.225 |
| agi_english/2172 | seed24762 | 0.225 |
| agi_english/2053 | seed24762 | 0.225 |
| agi_english/2060 | seed24762 | 0.225 |
| agi_english/2326 | seed24762 | 0.225 |
| agi_english/1131 | seed24762 | 0.225 |
| agi_english/1132 | seed24762 | 0.225 |
| agi_english/991 | seed24762 | 0.225 |
| agi_english/1786 | seed24762 | 0.225 |
| agi_english/981 | seed24762 | 0.225 |
| agi_english/978 | seed24762 | 0.225 |
| agi_english/959 | seed24762 | 0.225 |
| agi_english/927 | seed24762 | 0.225 |
| agi_english/868 | seed24762 | 0.225 |
| agi_english/912 | seed24762 | 0.225 |
| agi_english/1798 | seed24762 | 0.225 |
| agi_english/274 | seed24762 | 0.225 |
| agi_english/2322 | seed24762 | 0.225 |
| agi_english/2088 | seed24762 | 0.225 |
| agi_english/2167 | seed24762 | 0.225 |
| agi_english/2086 | seed24762 | 0.225 |
| agi_english/1119 | seed24762 | 0.225 |
| agi_english/1641 | seed24762 | 0.225 |
| agi_english/1862 | seed24762 | 0.225 |
| agi_english/2502 | seed24762 | 0.225 |
| agi_english/945 | seed24762 | 0.225 |
| agi_english/956 | seed24762 | 0.225 |
| agi_english/873 | seed24762 | 0.225 |
| agi_english/915 | seed24762 | 0.225 |
| agi_english/466 | seed24762 | 0.225 |
| agi_english/910 | seed24762 | 0.225 |
| agi_english/1124 | seed24762 | 0.225 |
| agi_english/1864 | seed24762 | 0.225 |
| agi_english/2523 | seed24762 | 0.225 |
| agi_english/1827 | seed24762 | 0.225 |
| agi_english/1832 | seed24762 | 0.225 |
| agi_english/731 | seed6529 | 0.213 |
| agi_english/17 | seed6529 | 0.213 |
| agi_english/1664 | seed6529 | 0.213 |
| agi_english/1663 | seed6529 | 0.213 |
| agi_english/1652 | seed6529 | 0.213 |
| agi_english/2052 | seed6529 | 0.213 |
| agi_english/802 | seed6529 | 0.213 |
| agi_english/79 | seed6529 | 0.213 |
| agi_english/1779 | seed6529 | 0.213 |
| agi_english/1034 | seed6529 | 0.213 |
| agi_english/1007 | seed6529 | 0.213 |
| agi_english/691 | seed6529 | 0.213 |
| agi_english/1714 | seed6529 | 0.213 |
| agi_english/2448 | seed6529 | 0.213 |
| agi_english/1711 | seed6529 | 0.213 |
| agi_english/163 | seed6529 | 0.213 |
| agi_english/160 | seed6529 | 0.213 |
| agi_english/1882 | seed6529 | 0.213 |
| agi_english/1835 | seed6529 | 0.213 |
| agi_english/1811 | seed6529 | 0.213 |
| agi_english/343 | seed6529 | 0.213 |
| agi_english/325 | seed6529 | 0.213 |
| agi_english/157 | seed6529 | 0.213 |
| agi_english/494 | seed6529 | 0.213 |
| agi_english/1972 | seed6529 | 0.213 |
| agi_english/48 | seed6529 | 0.213 |
| agi_english/392 | seed6529 | 0.213 |
| agi_english/953 | seed6529 | 0.213 |
| agi_english/93 | seed6529 | 0.213 |
| agi_english/61 | seed6529 | 0.213 |
| agi_english/630 | seed6529 | 0.213 |
| agi_english/1127 | seed6529 | 0.213 |
| agi_english/995 | seed6529 | 0.213 |
These are 10 problems with the lowest correlation with the overall evaluation (i.e. better models tend to do worse on these. )
| example_link | pass1_of_ex | tau |
|---|---|---|
| agi_english/2183 | 0.500 | -0.745 |
| agi_english/2430 | 0.600 | -0.730 |
| agi_english/1202 | 0.600 | -0.730 |
| agi_english/1448 | 0.600 | -0.730 |
| agi_english/2154 | 0.600 | -0.730 |
| agi_english/1543 | 0.600 | -0.730 |
| agi_english/2136 | 0.600 | -0.730 |
| agi_english/1081 | 0.300 | -0.683 |
| agi_english/570 | 0.500 | -0.626 |
| agi_english/1241 | 0.500 | -0.626 |
Histogram of problems by the accuracy on each problem.
Histogram of problems by the minimum win rate to solve each problem.