nq_step50000: by examples

Results Paper Code


Not solved by any model

There are 2426 examples not solved by any model. Solving some of these can be a good signal that your model is indeed better than leading models if these are good problems.
nq/0, nq/1, nq/10, nq/100, nq/1000, nq/1002, nq/1003, nq/1004, nq/1005, nq/1007, nq/1008, nq/101, nq/1011, nq/1012, nq/1013, nq/1014, nq/1015, nq/1016, nq/1017, nq/1018, nq/1019, nq/102, nq/1021, nq/1022, nq/1025, nq/1026, nq/1027, nq/1028, nq/1029, nq/103, nq/1032, nq/1033, nq/1035, nq/1036, nq/1037, nq/104, nq/1041, nq/1042, nq/1043, nq/1044, nq/1047, nq/1048, nq/105, nq/1050, nq/1052, nq/1053, nq/1054, nq/1058, nq/1059, nq/106, nq/1060, nq/1062, nq/1063, nq/1064, nq/1066, nq/1068, nq/1071, nq/1074, nq/1075, nq/1077, nq/1079, nq/108, nq/1082, nq/1083, nq/1084, nq/1085, nq/1089, nq/109, nq/1090, nq/1094, nq/1095, nq/1097, nq/1098, nq/1099, nq/110, nq/1100, nq/1102, nq/1104, nq/1108, nq/111, nq/1111, nq/1112, nq/1113, nq/1115, nq/1117, nq/1118, nq/1119, nq/112, nq/1123, nq/1128, nq/1129, nq/113, nq/1130, nq/1132, nq/1136, nq/1139, nq/114, nq/1141, nq/1143, nq/1145, nq/1146, nq/1149, nq/1150, nq/1151, nq/1152, nq/1154, nq/1156, nq/116, nq/1160, nq/1161, nq/1162, nq/1163, nq/1166, nq/1169, nq/1170, nq/1173, nq/1175, nq/1176, nq/1177, nq/118, nq/1180, nq/1181, nq/1182, nq/1186, nq/1187, nq/1189, nq/119, nq/1190, nq/1191, nq/1192, nq/1195, nq/1196, nq/1197, nq/1198, nq/120, nq/1200, nq/1201, nq/1202, nq/1203, nq/1205, nq/1208, nq/121, nq/1210, nq/1211, nq/1212, nq/1214, nq/1215, nq/1216, nq/1218, nq/122, nq/1220, nq/1221, nq/1222, nq/1223, nq/1224, nq/1227, nq/1229, nq/1230, nq/1231, nq/1233, nq/1234, nq/1235, nq/1236, nq/1237, nq/1238, nq/1239, nq/124, nq/1240, nq/1241, nq/1244, nq/1245, nq/1249, nq/125, nq/1252, nq/1253, nq/1254, nq/1255, nq/1257, nq/1258, nq/1259, nq/126, nq/1260, nq/1261, nq/1263, nq/1265, nq/1266, nq/1267, nq/1269, nq/127, nq/1271, nq/1272, nq/1273, nq/1274, nq/1279, nq/1281, nq/1283, nq/1284, nq/1288, nq/1290, nq/1291, nq/1298, nq/1299, nq/130, nq/1300, nq/1301, nq/1302, nq/1304, nq/1306, nq/1307, nq/1308, nq/131, nq/1310, nq/1311, nq/1312, nq/1313, nq/1314, nq/1319, nq/132, nq/1320, nq/1321, nq/1322, nq/1325, nq/1326, nq/1328, nq/1329, nq/133, nq/1331, nq/1332, nq/1334, nq/1336, nq/1337, nq/1338, nq/134, nq/1340, nq/1342, nq/1343, nq/1344, nq/1345, nq/1347, nq/1348, nq/1349, nq/135, nq/1351, nq/1353, nq/1354, nq/1358, nq/1359, nq/1360, nq/1362, nq/1363, nq/1365, nq/1366, nq/1367, nq/1368, nq/1370, nq/1372, nq/1373, nq/1374, nq/1375, nq/1376, nq/1377, nq/1378, nq/1379, nq/138, nq/1380, nq/1381, nq/1382, nq/1384, nq/1386, nq/1387, nq/139, nq/1391, nq/1392, nq/1393, nq/1395, nq/1396, nq/1398, nq/1399, nq/14, nq/140, nq/1400, nq/1401, nq/1402, nq/1403, nq/1404, nq/1405, nq/1406, nq/1407, nq/1410, nq/1411, nq/1412, nq/1414, nq/1417, nq/1418, nq/1419, nq/1420, nq/1421, nq/1422, nq/1423, nq/1425, nq/1426, nq/1427, nq/1428, nq/143, nq/1430, nq/1431, nq/1432, nq/1436, nq/1437, nq/1440, nq/1441, nq/1443, nq/1444, nq/1445, nq/1446, nq/1448, nq/1449, nq/1450, nq/1451, nq/1452, nq/1453, nq/1454, nq/1455, nq/1456, nq/1459, nq/146, nq/1461, nq/1462, nq/1463, nq/1464, nq/1465, nq/1466, nq/1467, nq/1468, nq/147, nq/1473, nq/1474, nq/1475, nq/1476, nq/1477, nq/1478, nq/1479, nq/148, nq/1480, nq/1481, nq/1482, nq/1483, nq/1484, nq/1485, nq/1487, nq/1488, nq/1491, nq/1492, nq/1493, nq/1494, nq/1495, nq/1496, nq/1498, nq/15, nq/150, nq/1500, nq/1502, nq/1503, nq/1504, nq/1505, nq/1506, nq/1507, nq/1509, nq/151, nq/1511, nq/1512, nq/1513, nq/1514, nq/1515, nq/1516, nq/1517, nq/1518, nq/152, nq/1520, nq/1521, nq/1522, nq/1525, nq/1526, nq/1528, nq/1529, nq/1530, nq/1532, nq/1533, nq/1534, nq/1536, nq/154, nq/1542, nq/1543, nq/1544, nq/1545, nq/1546, nq/1548, nq/1549, nq/155, nq/1550, nq/1553, nq/1554, nq/1556, nq/1559, nq/156, nq/1560, nq/1562, nq/1563, nq/1564, nq/1565, nq/1566, nq/1567, nq/1568, nq/157, nq/1572, nq/1573, nq/1574, nq/1575, nq/1577, nq/1578, nq/1579, nq/158, nq/1580, nq/1582, nq/1585, nq/1588, nq/1589, nq/1590, nq/1591, nq/1592, nq/1593, nq/1594, nq/1595, nq/1596, nq/1597, nq/160, nq/1601, nq/1602, nq/1603, nq/1604, nq/1605, nq/1606, nq/1607, nq/1608, nq/1609, nq/1612, nq/1613, nq/1614, nq/1615, nq/1617, nq/162, nq/1621, nq/1624, nq/1626, nq/1627, nq/1628, nq/163, nq/1631, nq/1632, nq/1633, nq/1634, nq/1636, nq/1637, nq/1638, nq/1639, nq/164, nq/1640, nq/1641, nq/1642, nq/1643, nq/1645, nq/1646, nq/1647, nq/1648, nq/165, nq/1650, nq/1651, nq/1652, nq/1653, nq/1654, nq/1656, nq/1657, nq/1658, nq/1659, nq/166, nq/1660, nq/1661, nq/1662, nq/1663, nq/1664, nq/1666, nq/1667, nq/1668, nq/1669, nq/167, nq/1671, nq/1672, nq/1673, nq/1675, nq/1676, nq/1677, nq/1678, nq/1681, nq/1682, nq/1683, nq/1684, nq/1685, nq/1687, nq/1689, nq/169, nq/1691, nq/1693, nq/1695, nq/1697, nq/1698, nq/17, nq/170, nq/1700, nq/1701, nq/1702, nq/1705, nq/1706, nq/1707, nq/1708, nq/1709, nq/1710, nq/1712, nq/1714, nq/1715, nq/1716, nq/1717, nq/1719, nq/172, nq/1722, nq/1723, nq/1724, nq/1725, nq/1726, nq/1727, nq/1728, nq/1729, nq/173, nq/1730, nq/1734, nq/1735, nq/1736, nq/1737, nq/1739, nq/174, nq/1742, nq/1743, nq/1745, nq/1746, nq/1749, nq/175, nq/1750, nq/1751, nq/1752, nq/1754, nq/1755, nq/1756, nq/1757, nq/1759, nq/176, nq/1762, nq/1763, nq/1764, nq/1765, nq/1766, nq/1767, nq/1768, nq/1770, nq/1773, nq/1774, nq/1776, nq/1778, nq/1779, nq/178, nq/1781, nq/1782, nq/1783, nq/1784, nq/1785, nq/1786, nq/1788, nq/179, nq/1791, nq/1793, nq/1794, nq/1795, nq/1796, nq/1798, nq/1801, nq/1802, nq/1804, nq/1805, nq/1806, nq/1807, nq/1808, nq/1809, nq/181, nq/1810, nq/1812, nq/1813, nq/1814, nq/1815, nq/1816, nq/1817, nq/1818, nq/1819, nq/1821, nq/1822, nq/1823, nq/1824, nq/1827, nq/1828, nq/1829, nq/183, nq/1830, nq/1833, nq/1834, nq/1836, nq/1837, nq/1838, nq/1839, nq/184, nq/1841, nq/1842, nq/1843, nq/1845, nq/1846, nq/1848, nq/1849, nq/1850, nq/1851, nq/1852, nq/1853, nq/1854, nq/1855, nq/1856, nq/1857, nq/1858, nq/1859, nq/1860, nq/1862, nq/1865, nq/1866, nq/1867, nq/1868, nq/1869, nq/187, nq/1870, nq/1872, nq/1873, nq/1874, nq/1875, nq/1876, nq/1877, nq/1878, nq/1880, nq/1882, nq/1883, nq/1886, nq/1887, nq/1888, nq/1889, nq/189, nq/1890, nq/1892, nq/1893, nq/1894, nq/1896, nq/1897, nq/1899, nq/19, nq/1900, nq/1901, nq/1902, nq/1903, nq/1906, nq/1907, nq/1908, nq/1909, nq/1910, nq/1911, nq/1912, nq/1913, nq/1915, nq/1916, nq/1917, nq/1918, nq/1919, nq/192, nq/1920, nq/1923, nq/1925, nq/1926, nq/193, nq/1930, nq/1933, nq/1934, nq/1935, nq/1936, nq/1937, nq/1938, nq/1939, nq/194, nq/1940, nq/1941, nq/1942, nq/1943, nq/1944, nq/1946, nq/1948, nq/1949, nq/1950, nq/1951, nq/1952, nq/1953, nq/1954, nq/1955, nq/1956, nq/1957, nq/1959, nq/196, nq/1960, nq/1961, nq/1964, nq/1965, nq/1967, nq/1968, nq/1969, nq/1971, nq/1972, nq/1973, nq/1977, nq/1979, nq/198, nq/1980, nq/1981, nq/1982, nq/1983, nq/1985, nq/1989, nq/199, nq/1992, nq/1993, nq/1997, nq/1998, nq/1999, nq/2, nq/200, nq/2000, nq/2001, nq/2002, nq/2003, nq/2004, nq/2008, nq/2009, nq/201, nq/2010, nq/2012, nq/2013, nq/2014, nq/2016, nq/2017, nq/2018, nq/202, nq/2020, nq/2022, nq/2023, nq/2024, nq/2029, nq/203, nq/2031, nq/2032, nq/2033, nq/2034, nq/2035, nq/2036, nq/2039, nq/204, nq/2042, nq/2043, nq/2044, nq/2045, nq/2046, nq/2047, nq/2049, nq/205, nq/2050, nq/2053, nq/2054, nq/2057, nq/2058, nq/2059, nq/206, nq/2060, nq/2063, nq/2065, nq/2066, nq/2067, nq/2068, nq/2070, nq/2072, nq/2073, nq/2074, nq/2076, nq/2077, nq/2079, nq/208, nq/2080, nq/2081, nq/2082, nq/2088, nq/2089, nq/209, nq/2090, nq/2092, nq/2093, nq/2094, nq/2096, nq/2097, nq/21, nq/2100, nq/2103, nq/2104, nq/2106, nq/2107, nq/2108, nq/2109, nq/211, nq/2112, nq/2113, nq/2114, nq/2116, nq/2117, nq/2118, nq/2119, nq/212, nq/2121, nq/2122, nq/2123, nq/2124, nq/2126, nq/2128, nq/2129, nq/213, nq/2131, nq/2133, nq/2134, nq/2135, nq/2136, nq/2137, nq/2139, nq/214, nq/2140, nq/2141, nq/2142, nq/2143, nq/2144, nq/2145, nq/2146, nq/2147, nq/2148, nq/215, nq/2150, nq/2151, nq/2154, nq/2155, nq/2156, nq/2157, nq/2158, nq/2159, nq/2160, nq/2162, nq/2166, nq/2169, nq/2172, nq/2173, nq/2176, nq/2177, nq/2179, nq/218, nq/2180, nq/2183, nq/2184, nq/2188, nq/2189, nq/2190, nq/2191, nq/2192, nq/2194, nq/2195, nq/2197, nq/220, nq/2200, nq/2201, nq/2202, nq/2203, nq/2204, nq/2205, nq/2206, nq/2207, nq/2208, nq/2209, nq/221, nq/2210, nq/2211, nq/2212, nq/2214, nq/2215, nq/2216, nq/2217, nq/2219, nq/222, nq/2220, nq/2223, nq/2224, nq/2225, nq/2227, nq/2228, nq/223, nq/2230, nq/2231, nq/2233, nq/2236, nq/2237, nq/2238, nq/2239, nq/224, nq/2241, nq/2242, nq/2243, nq/2244, nq/2245, nq/2246, nq/2248, nq/225, nq/2250, nq/2252, nq/2256, nq/2258, nq/2259, nq/2261, nq/2262, nq/2263, nq/2264, nq/2265, nq/2267, nq/2268, nq/227, nq/2271, nq/2272, nq/2275, nq/2276, nq/2278, nq/2279, nq/2280, nq/2281, nq/2282, nq/2284, nq/2285, nq/2286, nq/2288, nq/229, nq/2290, nq/2291, nq/2292, nq/2293, nq/2294, nq/2295, nq/2298, nq/2302, nq/2304, nq/2305, nq/2307, nq/2308, nq/2309, nq/2310, nq/2311, nq/2312, nq/2313, nq/2314, nq/2315, nq/2318, nq/2319, nq/232, nq/2322, nq/2323, nq/2324, nq/2328, nq/2329, nq/233, nq/2330, nq/2331, nq/2334, nq/2338, nq/2339, nq/234, nq/2340, nq/2342, nq/2343, nq/2346, nq/2348, nq/2349, nq/2354, nq/2355, nq/2356, nq/2357, nq/2358, nq/2359, nq/2360, nq/2361, nq/2364, nq/2365, nq/2366, nq/2367, nq/237, nq/2370, nq/2371, nq/2372, nq/2373, nq/2374, nq/2375, nq/2376, nq/2378, nq/2379, nq/238, nq/2380, nq/2382, nq/2383, nq/2384, nq/2386, nq/2388, nq/2389, nq/2390, nq/2391, nq/2393, nq/2395, nq/2397, nq/24, nq/240, nq/2400, nq/2401, nq/2402, nq/2403, nq/2405, nq/2408, nq/2409, nq/2410, nq/2411, nq/2414, nq/2417, nq/2418, nq/242, nq/2421, nq/2422, nq/2423, nq/2424, nq/2426, nq/243, nq/2430, nq/2433, nq/2434, nq/2435, nq/2437, nq/2440, nq/2441, nq/2445, nq/2446, nq/2447, nq/2448, nq/2449, nq/245, nq/2450, nq/2451, nq/2452, nq/2453, nq/2455, nq/2456, nq/2458, nq/2459, nq/2460, nq/2462, nq/2464, nq/2465, nq/2466, nq/2467, nq/2469, nq/247, nq/2470, nq/2471, nq/2472, nq/2473, nq/2477, nq/2478, nq/248, nq/2480, nq/2481, nq/2482, nq/2483, nq/2485, nq/2487, nq/2488, nq/2489, nq/2490, nq/2491, nq/2493, nq/2494, nq/2496, nq/2498, nq/2499, nq/25, nq/2500, nq/2501, nq/2502, nq/2503, nq/2506, nq/2507, nq/2508, nq/2509, nq/251, nq/2510, nq/2511, nq/2512, nq/2513, nq/2517, nq/2519, nq/252, nq/2520, nq/2523, nq/2524, nq/2525, nq/2526, nq/2528, nq/2529, nq/253, nq/2530, nq/2531, nq/2532, nq/2533, nq/2536, nq/2539, nq/2541, nq/2542, nq/2543, nq/2545, nq/2548, nq/2549, nq/255, nq/2550, nq/2553, nq/2554, nq/2555, nq/2557, nq/2559, nq/2560, nq/2562, nq/2563, nq/2565, nq/2566, nq/2568, nq/2569, nq/257, nq/2573, nq/2574, nq/2576, nq/2577, nq/2578, nq/2579, nq/258, nq/2580, nq/2582, nq/2585, nq/2586, nq/2588, nq/2589, nq/259, nq/2591, nq/2593, nq/2594, nq/2595, nq/2597, nq/2599, nq/260, nq/2601, nq/2602, nq/2603, nq/2604, nq/2605, nq/2606, nq/2607, nq/2608, nq/2611, nq/2613, nq/2614, nq/2615, nq/2617, nq/2618, nq/2619, nq/262, nq/2620, nq/2622, nq/2623, nq/2624, nq/2627, nq/2628, nq/2629, nq/263, nq/2631, nq/2632, nq/2633, nq/2636, nq/2637, nq/2638, nq/2643, nq/2644, nq/2645, nq/2648, nq/2649, nq/265, nq/2651, nq/2652, nq/2654, nq/2655, nq/2656, nq/2657, nq/2658, nq/2659, nq/266, nq/2660, nq/2661, nq/2662, nq/2664, nq/2669, nq/267, nq/2670, nq/2671, nq/2672, nq/2673, nq/2674, nq/2675, nq/2676, nq/2677, nq/2678, nq/268, nq/2681, nq/2685, nq/2686, nq/2687, nq/2688, nq/269, nq/2690, nq/2693, nq/2694, nq/2695, nq/2696, nq/2699, nq/270, nq/2701, nq/2702, nq/2703, nq/2706, nq/2707, nq/2708, nq/2709, nq/271, nq/2710, nq/2711, nq/2712, nq/2715, nq/2716, nq/2717, nq/2718, nq/2719, nq/272, nq/2720, nq/2722, nq/2724, nq/2725, nq/2726, nq/2727, nq/2729, nq/273, nq/2733, nq/2734, nq/2735, nq/2737, nq/2738, nq/2739, nq/2740, nq/2742, nq/2743, nq/2744, nq/2745, nq/2747, nq/2748, nq/275, nq/2750, nq/2751, nq/2752, nq/2754, nq/2756, nq/2757, nq/2758, nq/276, nq/2763, nq/2764, nq/2765, nq/2766, nq/2767, nq/2768, nq/2771, nq/2773, nq/2774, nq/2775, nq/2778, nq/278, nq/2782, nq/2783, nq/2785, nq/2786, nq/2787, nq/2788, nq/2789, nq/279, nq/2792, nq/2793, nq/2794, nq/2795, nq/2796, nq/2797, nq/2798, nq/2799, nq/28, nq/2800, nq/2801, nq/2802, nq/2803, nq/2804, nq/2805, nq/2807, nq/2809, nq/2812, nq/2813, nq/2814, nq/2819, nq/282, nq/2821, nq/2822, nq/2823, nq/2827, nq/2828, nq/283, nq/2830, nq/2831, nq/2832, nq/2833, nq/2838, nq/2839, nq/2840, nq/2841, nq/2842, nq/2844, nq/2845, nq/2846, nq/2847, nq/2848, nq/2849, nq/2850, nq/2851, nq/2855, nq/2856, nq/2857, nq/2858, nq/2859, nq/286, nq/2860, nq/2861, nq/2862, nq/2863, nq/2864, nq/2865, nq/2866, nq/2868, nq/2869, nq/287, nq/2871, nq/2872, nq/2875, nq/2876, nq/2877, nq/2878, nq/2879, nq/288, nq/2881, nq/2882, nq/2883, nq/2884, nq/2885, nq/2886, nq/2887, nq/2889, nq/289, nq/2890, nq/2893, nq/2894, nq/2895, nq/2897, nq/29, nq/290, nq/2900, nq/2901, nq/2902, nq/2903, nq/2904, nq/2905, nq/2907, nq/2908, nq/291, nq/2911, nq/2912, nq/2913, nq/2914, nq/2916, nq/2918, nq/292, nq/2920, nq/2923, nq/2924, nq/2927, nq/2928, nq/2929, nq/2930, nq/2931, nq/2932, nq/2933, nq/2934, nq/2935, nq/2936, nq/2937, nq/2939, nq/294, nq/2940, nq/2941, nq/2942, nq/2944, nq/2945, nq/2947, nq/295, nq/2950, nq/2952, nq/2954, nq/2955, nq/2957, nq/2958, nq/2961, nq/2962, nq/2963, nq/2965, nq/2966, nq/2967, nq/2968, nq/297, nq/2971, nq/2972, nq/2973, nq/2975, nq/2977, nq/2978, nq/2979, nq/2980, nq/2982, nq/2983, nq/2984, nq/2985, nq/2987, nq/2988, nq/299, nq/2990, nq/2992, nq/2993, nq/2994, nq/2995, nq/2996, nq/2998, nq/2999, nq/3, nq/300, nq/3002, nq/3003, nq/3004, nq/3006, nq/3007, nq/3008, nq/3009, nq/301, nq/3010, nq/3011, nq/3013, nq/3014, nq/302, nq/3022, nq/3025, nq/3027, nq/3028, nq/3029, nq/303, nq/3030, nq/3031, nq/3032, nq/3033, nq/3034, nq/3035, nq/3036, nq/3037, nq/3038, nq/3039, nq/3041, nq/3042, nq/3043, nq/3044, nq/3046, nq/3047, nq/3048, nq/3049, nq/3053, nq/3054, nq/3055, nq/3056, nq/3059, nq/3060, nq/3063, nq/3064, nq/3066, nq/3067, nq/3068, nq/307, nq/3070, nq/3071, nq/3072, nq/3073, nq/3074, nq/3075, nq/3076, nq/3077, nq/308, nq/3080, nq/3081, nq/3083, nq/3084, nq/3085, nq/3088, nq/3089, nq/309, nq/3093, nq/3094, nq/3095, nq/3096, nq/3097, nq/3098, nq/3099, nq/310, nq/3100, nq/3102, nq/3104, nq/3106, nq/3107, nq/311, nq/3110, nq/3111, nq/3112, nq/3113, nq/3114, nq/3115, nq/3117, nq/3118, nq/3119, nq/312, nq/3120, nq/3121, nq/3122, nq/3124, nq/3125, nq/3126, nq/3127, nq/3128, nq/3129, nq/313, nq/3130, nq/3131, nq/3134, nq/3135, nq/3136, nq/3138, nq/314, nq/3141, nq/3142, nq/3143, nq/3144, nq/3145, nq/3146, nq/3147, nq/3148, nq/3149, nq/315, nq/3150, nq/3151, nq/3153, nq/3154, nq/3155, nq/3157, nq/3159, nq/316, nq/3160, nq/3161, nq/3162, nq/3165, nq/3166, nq/3169, nq/3170, nq/3171, nq/3173, nq/3174, nq/3175, nq/3176, nq/318, nq/3180, nq/3181, nq/3183, nq/3184, nq/3186, nq/3188, nq/3189, nq/319, nq/3190, nq/3191, nq/3192, nq/3193, nq/3194, nq/3196, nq/3197, nq/3198, nq/3199, nq/32, nq/3200, nq/3202, nq/3204, nq/3205, nq/3207, nq/3208, nq/3209, nq/3210, nq/3212, nq/3213, nq/3215, nq/3216, nq/3218, nq/322, nq/3220, nq/3224, nq/3227, nq/3230, nq/3232, nq/3234, nq/3235, nq/3236, nq/3237, nq/3238, nq/324, nq/3242, nq/3243, nq/3244, nq/3245, nq/3246, nq/3247, nq/3248, nq/325, nq/3252, nq/3254, nq/3255, nq/3256, nq/3259, nq/326, nq/3260, nq/3261, nq/3262, nq/3263, nq/3264, nq/3267, nq/3269, nq/3270, nq/3271, nq/3273, nq/3274, nq/3275, nq/3276, nq/3277, nq/3278, nq/328, nq/3281, nq/3283, nq/3284, nq/3285, nq/3286, nq/3287, nq/3289, nq/329, nq/3290, nq/3291, nq/3292, nq/3295, nq/3297, nq/3299, nq/3300, nq/3301, nq/3302, nq/3303, nq/3304, nq/3307, nq/3310, nq/3311, nq/3312, nq/3313, nq/3314, nq/3316, nq/3317, nq/3318, nq/3319, nq/332, nq/3320, nq/3321, nq/3322, nq/3324, nq/3325, nq/3326, nq/3328, nq/3329, nq/333, nq/3330, nq/3331, nq/3334, nq/3336, nq/3337, nq/3339, nq/3344, nq/3345, nq/3347, nq/3350, nq/3351, nq/3352, nq/3353, nq/3354, nq/3356, nq/3359, nq/3360, nq/3361, nq/3362, nq/3364, nq/3368, nq/3369, nq/337, nq/3370, nq/3371, nq/3374, nq/3375, nq/3376, nq/3378, nq/3379, nq/3380, nq/3381, nq/3382, nq/3383, nq/3388, nq/3389, nq/339, nq/3390, nq/3391, nq/3392, nq/3393, nq/3396, nq/3397, nq/3398, nq/3399, nq/340, nq/3400, nq/3402, nq/3403, nq/3404, nq/3405, nq/3406, nq/3408, nq/3409, nq/3410, nq/3411, nq/3412, nq/3415, nq/3416, nq/3417, nq/3418, nq/3419, nq/342, nq/3420, nq/3421, nq/3423, nq/3425, nq/3428, nq/3429, nq/343, nq/3430, nq/3431, nq/3432, nq/3433, nq/3434, nq/3435, nq/3437, nq/3438, nq/3439, nq/344, nq/3440, nq/3444, nq/3445, nq/3446, nq/3447, nq/3448, nq/3449, nq/345, nq/3450, nq/3452, nq/3453, nq/3455, nq/3456, nq/3459, nq/346, nq/3460, nq/3461, nq/3462, nq/3464, nq/3465, nq/347, nq/3470, nq/3471, nq/3472, nq/3475, nq/3476, nq/3477, nq/3478, nq/3479, nq/3480, nq/3486, nq/3489, nq/3490, nq/3491, nq/3493, nq/3494, nq/3496, nq/3498, nq/3499, nq/35, nq/3500, nq/3501, nq/3503, nq/3504, nq/3505, nq/3507, nq/3508, nq/3509, nq/351, nq/3510, nq/3511, nq/3513, nq/3514, nq/3515, nq/3516, nq/3518, nq/3519, nq/352, nq/3521, nq/3522, nq/3523, nq/3525, nq/353, nq/3532, nq/3533, nq/3534, nq/3535, nq/3536, nq/3538, nq/3539, nq/354, nq/3540, nq/3541, nq/3542, nq/3543, nq/3544, nq/3545, nq/3547, nq/3548, nq/3549, nq/3551, nq/3552, nq/3553, nq/3554, nq/3555, nq/3558, nq/3559, nq/356, nq/3561, nq/3562, nq/3563, nq/3566, nq/3567, nq/3568, nq/357, nq/3572, nq/3573, nq/3575, nq/3576, nq/3577, nq/3579, nq/3580, nq/3581, nq/3582, nq/3583, nq/3584, nq/3585, nq/3586, nq/3587, nq/3589, nq/3592, nq/3594, nq/3595, nq/3596, nq/3597, nq/3598, nq/3599, nq/36, nq/3601, nq/3602, nq/3603, nq/3604, nq/3606, nq/3608, nq/363, nq/364, nq/366, nq/368, nq/37, nq/370, nq/372, nq/373, nq/374, nq/376, nq/377, nq/378, nq/38, nq/380, nq/381, nq/383, nq/384, nq/385, nq/386, nq/388, nq/389, nq/39, nq/390, nq/391, nq/392, nq/393, nq/394, nq/395, nq/399, nq/4, nq/40, nq/400, nq/401, nq/402, nq/404, nq/405, nq/407, nq/408, nq/409, nq/41, nq/410, nq/411, nq/413, nq/414, nq/416, nq/419, nq/42, nq/421, nq/422, nq/423, nq/424, nq/425, nq/428, nq/429, nq/432, nq/433, nq/434, nq/436, nq/438, nq/439, nq/44, nq/440, nq/441, nq/442, nq/443, nq/444, nq/446, nq/447, nq/448, nq/449, nq/452, nq/453, nq/454, nq/455, nq/456, nq/458, nq/459, nq/46, nq/461, nq/464, nq/465, nq/466, nq/467, nq/469, nq/470, nq/471, nq/473, nq/474, nq/476, nq/477, nq/478, nq/479, nq/48, nq/482, nq/484, nq/485, nq/486, nq/488, nq/489, nq/49, nq/490, nq/491, nq/493, nq/495, nq/496, nq/497, nq/498, nq/499, nq/5, nq/50, nq/500, nq/501, nq/503, nq/504, nq/505, nq/507, nq/508, nq/509, nq/51, nq/510, nq/511, nq/512, nq/513, nq/514, nq/515, nq/517, nq/518, nq/519, nq/52, nq/520, nq/522, nq/524, nq/525, nq/528, nq/529, nq/53, nq/532, nq/535, nq/536, nq/537, nq/538, nq/539, nq/540, nq/542, nq/545, nq/546, nq/547, nq/549, nq/556, nq/557, nq/558, nq/559, nq/56, nq/560, nq/561, nq/563, nq/565, nq/566, nq/567, nq/568, nq/569, nq/57, nq/570, nq/572, nq/573, nq/576, nq/578, nq/579, nq/58, nq/580, nq/583, nq/584, nq/586, nq/587, nq/589, nq/590, nq/591, nq/592, nq/594, nq/596, nq/597, nq/598, nq/599, nq/6, nq/60, nq/600, nq/602, nq/604, nq/605, nq/609, nq/61, nq/611, nq/612, nq/613, nq/614, nq/616, nq/618, nq/62, nq/620, nq/621, nq/623, nq/624, nq/626, nq/627, nq/629, nq/631, nq/633, nq/634, nq/636, nq/637, nq/639, nq/64, nq/640, nq/641, nq/644, nq/645, nq/646, nq/647, nq/648, nq/649, nq/652, nq/655, nq/657, nq/658, nq/660, nq/662, nq/663, nq/664, nq/666, nq/668, nq/669, nq/67, nq/670, nq/672, nq/673, nq/676, nq/677, nq/678, nq/68, nq/680, nq/682, nq/683, nq/684, nq/686, nq/69, nq/690, nq/691, nq/693, nq/695, nq/696, nq/697, nq/698, nq/699, nq/7, nq/70, nq/700, nq/701, nq/702, nq/704, nq/707, nq/709, nq/71, nq/710, nq/711, nq/714, nq/715, nq/717, nq/718, nq/719, nq/72, nq/720, nq/722, nq/723, nq/724, nq/726, nq/727, nq/728, nq/73, nq/732, nq/734, nq/735, nq/738, nq/74, nq/742, nq/743, nq/745, nq/746, nq/747, nq/749, nq/75, nq/750, nq/751, nq/752, nq/753, nq/755, nq/756, nq/757, nq/758, nq/76, nq/760, nq/761, nq/762, nq/763, nq/764, nq/765, nq/766, nq/767, nq/768, nq/77, nq/770, nq/771, nq/772, nq/775, nq/776, nq/777, nq/778, nq/779, nq/780, nq/781, nq/782, nq/784, nq/785, nq/786, nq/788, nq/79, nq/790, nq/791, nq/792, nq/793, nq/794, nq/795, nq/796, nq/80, nq/800, nq/801, nq/803, nq/804, nq/805, nq/806, nq/807, nq/808, nq/81, nq/811, nq/812, nq/813, nq/814, nq/817, nq/818, nq/819, nq/82, nq/822, nq/823, nq/824, nq/827, nq/828, nq/829, nq/83, nq/830, nq/831, nq/832, nq/834, nq/837, nq/838, nq/839, nq/84, nq/840, nq/842, nq/844, nq/845, nq/848, nq/849, nq/85, nq/851, nq/853, nq/858, nq/859, nq/86, nq/860, nq/863, nq/864, nq/865, nq/867, nq/869, nq/87, nq/870, nq/872, nq/873, nq/876, nq/877, nq/878, nq/879, nq/88, nq/882, nq/883, nq/884, nq/885, nq/886, nq/887, nq/889, nq/89, nq/890, nq/891, nq/892, nq/893, nq/894, nq/896, nq/897, nq/899, nq/9, nq/90, nq/900, nq/901, nq/903, nq/904, nq/905, nq/907, nq/91, nq/911, nq/913, nq/915, nq/916, nq/917, nq/919, nq/920, nq/921, nq/922, nq/926, nq/928, nq/930, nq/931, nq/932, nq/933, nq/938, nq/939, nq/94, nq/941, nq/942, nq/943, nq/945, nq/946, nq/947, nq/948, nq/949, nq/95, nq/950, nq/951, nq/953, nq/955, nq/957, nq/958, nq/959, nq/960, nq/961, nq/962, nq/963, nq/964, nq/965, nq/966, nq/968, nq/969, nq/97, nq/971, nq/972, nq/974, nq/975, nq/976, nq/977, nq/978, nq/98, nq/981, nq/984, nq/986, nq/988, nq/990, nq/991, nq/993, nq/994, nq/995, nq/996, nq/997, nq/998, nq/999

Problems solved by 1 model only

example_link model min_pass1_of_model
nq/1024 seed17495 0.178
nq/1126 seed17495 0.178
nq/12 seed17495 0.178
nq/1295 seed17495 0.178
nq/1293 seed17495 0.178
nq/129 seed17495 0.178
nq/1316 seed17495 0.178
nq/1510 seed17495 0.178
nq/1731 seed17495 0.178
nq/1620 seed17495 0.178
nq/1861 seed17495 0.178
nq/1789 seed17495 0.178
nq/1761 seed17495 0.178
nq/1741 seed17495 0.178
nq/2350 seed17495 0.178
nq/1988 seed17495 0.178
nq/2019 seed17495 0.178
nq/1996 seed17495 0.178
nq/2443 seed17495 0.178
nq/462 seed17495 0.178
nq/387 seed17495 0.178
nq/406 seed17495 0.178
nq/3309 seed17495 0.178
nq/3061 seed17495 0.178
nq/3298 seed17495 0.178
nq/3185 seed17495 0.178
nq/2835 seed17495 0.178
nq/2880 seed17495 0.178
nq/2680 seed17495 0.178
nq/2682 seed17495 0.178
nq/96 seed17495 0.178
nq/2168 seed27712 0.173
nq/2377 seed27712 0.173
nq/2235 seed27712 0.173
nq/1792 seed27712 0.173
nq/230 seed27712 0.173
nq/235 seed27712 0.173
nq/2234 seed27712 0.173
nq/675 seed27712 0.173
nq/862 seed27712 0.173
nq/2760 seed27712 0.173
nq/3005 seed27712 0.173
nq/3172 seed27712 0.173
nq/431 seed27712 0.173
nq/744 seed27712 0.173
nq/502 seed27712 0.173
nq/2639 seed27712 0.173
nq/3201 seed27712 0.173
nq/3103 seed27712 0.173
nq/3219 seed27712 0.173
nq/23 seed27712 0.173
nq/2161 seed27712 0.173
nq/1871 seed27712 0.173
nq/210 seed27712 0.173
nq/1317 seed27712 0.173
nq/1335 seed27712 0.173
nq/2289 seed27712 0.173
nq/188 seed27712 0.173
nq/1137 seed27712 0.173
nq/123 seed27712 0.173
nq/1101 seed27712 0.173
nq/1010 seed27712 0.173
nq/2635 seed27712 0.173
nq/2728 seed30841 0.173
nq/3231 seed30841 0.173
nq/280 seed30841 0.173
nq/2870 seed30841 0.173
nq/2834 seed30841 0.173
nq/2650 seed30841 0.173
nq/2642 seed30841 0.173
nq/2187 seed30841 0.173
nq/2534 seed30841 0.173
nq/1286 seed30841 0.173
nq/1116 seed30841 0.173
nq/1576 seed30841 0.173
nq/1686 seed30841 0.173
nq/1538 seed30841 0.173
nq/2048 seed30841 0.173
nq/1155 seed30841 0.173
nq/909 seed30841 0.173
nq/856 seed30841 0.173
nq/850 seed30841 0.173
nq/66 seed30841 0.173
nq/585 seed30841 0.173
nq/397 seed30841 0.173
nq/3524 seed30841 0.173
nq/2991 seed16385 0.168
nq/2646 seed16385 0.168
nq/3414 seed16385 0.168
nq/3591 seed16385 0.168
nq/3484 seed16385 0.168
nq/3338 seed16385 0.168
nq/642 seed16385 0.168
nq/379 seed16385 0.168
nq/1611 seed16385 0.168
nq/115 seed16385 0.168
nq/2226 seed16385 0.168
nq/1213 seed16385 0.168
nq/1932 seed16385 0.168
nq/2127 seed16385 0.168
nq/2005 seed16385 0.168
nq/1975 seed16385 0.168
nq/2546 seed149 0.168
nq/3401 seed149 0.168
nq/741 seed149 0.168
nq/671 seed149 0.168
nq/3407 seed149 0.168
nq/445 seed149 0.168
nq/3296 seed149 0.168
nq/2917 seed149 0.168
nq/875 seed149 0.168
nq/2232 seed149 0.168
nq/197 seed149 0.168
nq/2062 seed149 0.168
nq/3012 seed149 0.168
nq/3163 seed149 0.168
nq/2836 seed149 0.168
nq/3222 seed149 0.168
nq/774 seed149 0.168
nq/16 seed149 0.168
nq/1439 seed149 0.168
nq/1080 seed149 0.168
nq/1140 seed149 0.168
nq/1124 seed149 0.168
nq/1020 seed149 0.168
nq/1091 seed149 0.168
nq/2906 seed149 0.168
nq/367 seed24762 0.168
nq/880 seed24762 0.168
nq/595 seed24762 0.168
nq/427 seed24762 0.168
nq/3052 seed24762 0.168
nq/430 seed24762 0.168
nq/1847 seed24762 0.168
nq/2518 seed24762 0.168
nq/2171 seed24762 0.168
nq/2317 seed24762 0.168
nq/241 seed24762 0.168
nq/3395 seed24762 0.168
nq/1039 seed24762 0.168
nq/1268 seed24762 0.168
nq/1297 seed24762 0.168
nq/2254 seed24762 0.168
nq/2193 seed24762 0.168
nq/1720 seed24762 0.168
nq/1397 seed24762 0.168
nq/1364 seed24762 0.168
nq/1489 seed24762 0.168
nq/1540 seed24762 0.168
nq/2852 seed24762 0.168
nq/2684 seed24762 0.168
nq/261 seed24762 0.168
nq/54 seed6529 0.164
nq/3590 seed6529 0.164
nq/3132 seed6529 0.164
nq/601 seed6529 0.164
nq/3482 seed6529 0.164
nq/888 seed6529 0.164
nq/3017 seed6529 0.164
nq/3203 seed6529 0.164
nq/33 seed6529 0.164
nq/2167 seed6529 0.164
nq/219 seed6529 0.164
nq/249 seed6529 0.164
nq/1844 seed6529 0.164
nq/2416 seed6529 0.164
nq/1339 seed6529 0.164
nq/1471 seed6529 0.164
nq/1246 seed6529 0.164
nq/1369 seed6529 0.164
nq/1947 seed6529 0.164
nq/92 seed6529 0.164
nq/802 seed6529 0.164
nq/729 seed6529 0.164
nq/2970 seed6529 0.164
nq/3105 seed6529 0.164
nq/3293 seed6529 0.164
nq/1138 seed6115 0.159
nq/1629 seed6115 0.159
nq/1065 seed6115 0.159
nq/2700 seed6115 0.159
nq/1243 seed6115 0.159
nq/1963 seed6115 0.159
nq/1390 seed6115 0.159
nq/1303 seed6115 0.159
nq/1356 seed6115 0.159
nq/2497 seed6115 0.159
nq/2341 seed6115 0.159
nq/254 seed6115 0.159
nq/244 seed6115 0.159
nq/927 seed6115 0.159
nq/855 seed6115 0.159
nq/712 seed6115 0.159
nq/833 seed6115 0.159
nq/3469 seed6115 0.159
nq/3385 seed6115 0.159
nq/480 seed6115 0.159
nq/2626 seed6115 0.159
nq/3078 seed6115 0.159
nq/3057 seed6115 0.159
nq/304 seed6115 0.159
nq/2826 seed6115 0.159
nq/3569 seed6115 0.159
nq/3221 seed6115 0.159
nq/3463 seed14309 0.147
nq/330 seed14309 0.147
nq/1318 seed14309 0.147
nq/1524 seed14309 0.147
nq/171 seed14309 0.147
nq/1167 seed14309 0.147
nq/1105 seed14309 0.147
nq/1076 seed14309 0.147
nq/1006 seed14309 0.147
nq/3467 seed14309 0.147
nq/2736 seed14309 0.147
nq/2770 seed14309 0.147
nq/2369 seed14309 0.147
nq/1780 seed14309 0.147
nq/2085 seed14309 0.147
nq/2584 seed14309 0.147
nq/2468 seed14309 0.147
nq/2535 seed14309 0.147
nq/769 seed14309 0.147
nq/679 seed14309 0.147
nq/564 seed14309 0.147
nq/494 seed14309 0.147
nq/3156 seed14309 0.147
nq/3019 seed14309 0.147
nq/3288 seed14309 0.147
nq/3195 seed14309 0.147
nq/2948 seed14309 0.147
nq/2951 seed14309 0.147
nq/1207 seed24734 0.144
nq/1157 seed24734 0.144
nq/1864 seed24734 0.144
nq/1924 seed24734 0.144
nq/1001 seed24734 0.144
nq/161 seed24734 0.144
nq/2527 seed24734 0.144
nq/2484 seed24734 0.144
nq/2273 seed24734 0.144
nq/2175 seed24734 0.144
nq/2714 seed24734 0.144
nq/2691 seed24734 0.144
nq/2634 seed24734 0.144
nq/2621 seed24734 0.144
nq/3280 seed24734 0.144
nq/3211 seed24734 0.144
nq/3139 seed24734 0.144
nq/3564 seed24734 0.144
nq/534 seed24734 0.144
nq/581 seed24734 0.144
nq/816 seed24734 0.144
nq/954 seed24734 0.144

Suspect problems

These are 10 problems with the lowest correlation with the overall evaluation (i.e. better models tend to do worse on these. )

example_link pass1_of_ex tau
nq/3600 0.300 -0.683
nq/1738 0.300 -0.618
nq/1799 0.800 -0.596
nq/2299 0.200 -0.596
nq/730 0.200 -0.596
nq/1825 0.500 -0.566
nq/306 0.700 -0.553
nq/2362 0.700 -0.553
nq/1732 0.400 -0.548
nq/3474 0.200 -0.522

Histogram of accuracies

Histogram of problems by the accuracy on each problem.

Histogram of difficulties

Histogram of problems by the minimum win rate to solve each problem.