hellaswag_step50000: by examples

Results Paper Code


Not solved by any model

There are 2103 examples not solved by any model. Solving some of these can be a good signal that your model is indeed better than leading models if these are good problems.
hellaswag/100, hellaswag/10001, hellaswag/10002, hellaswag/10004, hellaswag/10009, hellaswag/10011, hellaswag/10020, hellaswag/10022, hellaswag/10026, hellaswag/10027, hellaswag/10029, hellaswag/1003, hellaswag/10030, hellaswag/10032, hellaswag/10038, hellaswag/1005, hellaswag/101, hellaswag/1010, hellaswag/1012, hellaswag/1015, hellaswag/1016, hellaswag/102, hellaswag/1022, hellaswag/1026, hellaswag/1029, hellaswag/103, hellaswag/1032, hellaswag/1038, hellaswag/1055, hellaswag/1059, hellaswag/1063, hellaswag/1067, hellaswag/1071, hellaswag/1073, hellaswag/1074, hellaswag/1075, hellaswag/1078, hellaswag/1082, hellaswag/1085, hellaswag/1087, hellaswag/1088, hellaswag/109, hellaswag/11, hellaswag/110, hellaswag/1100, hellaswag/1108, hellaswag/1109, hellaswag/1112, hellaswag/1115, hellaswag/1125, hellaswag/1140, hellaswag/1146, hellaswag/1151, hellaswag/1164, hellaswag/1165, hellaswag/1168, hellaswag/1176, hellaswag/1177, hellaswag/1178, hellaswag/118, hellaswag/1181, hellaswag/1184, hellaswag/12, hellaswag/1200, hellaswag/1202, hellaswag/1205, hellaswag/1210, hellaswag/1211, hellaswag/1212, hellaswag/1213, hellaswag/1215, hellaswag/1216, hellaswag/1217, hellaswag/1224, hellaswag/1227, hellaswag/123, hellaswag/1235, hellaswag/1237, hellaswag/1239, hellaswag/124, hellaswag/1244, hellaswag/1245, hellaswag/1247, hellaswag/1249, hellaswag/125, hellaswag/1261, hellaswag/1265, hellaswag/1266, hellaswag/1271, hellaswag/1275, hellaswag/1280, hellaswag/1282, hellaswag/1285, hellaswag/1287, hellaswag/1291, hellaswag/1294, hellaswag/1304, hellaswag/131, hellaswag/1315, hellaswag/1316, hellaswag/1337, hellaswag/1342, hellaswag/1345, hellaswag/1354, hellaswag/1356, hellaswag/1358, hellaswag/1364, hellaswag/1365, hellaswag/1368, hellaswag/1375, hellaswag/1376, hellaswag/1379, hellaswag/1383, hellaswag/1389, hellaswag/1390, hellaswag/1395, hellaswag/1397, hellaswag/1400, hellaswag/1401, hellaswag/1405, hellaswag/1407, hellaswag/1409, hellaswag/1410, hellaswag/1412, hellaswag/1414, hellaswag/142, hellaswag/1420, hellaswag/1421, hellaswag/1424, hellaswag/1426, hellaswag/1434, hellaswag/1438, hellaswag/1444, hellaswag/1446, hellaswag/1448, hellaswag/1449, hellaswag/145, hellaswag/1452, hellaswag/1458, hellaswag/1460, hellaswag/1465, hellaswag/1467, hellaswag/1468, hellaswag/1469, hellaswag/1474, hellaswag/1476, hellaswag/1477, hellaswag/148, hellaswag/1480, hellaswag/1482, hellaswag/1487, hellaswag/1494, hellaswag/1497, hellaswag/1502, hellaswag/1506, hellaswag/1507, hellaswag/1508, hellaswag/1509, hellaswag/151, hellaswag/1510, hellaswag/1512, hellaswag/1515, hellaswag/1516, hellaswag/152, hellaswag/1525, hellaswag/1529, hellaswag/153, hellaswag/1532, hellaswag/1536, hellaswag/1540, hellaswag/1541, hellaswag/1542, hellaswag/1543, hellaswag/1544, hellaswag/1545, hellaswag/1547, hellaswag/1550, hellaswag/1553, hellaswag/1554, hellaswag/1555, hellaswag/1556, hellaswag/1565, hellaswag/1567, hellaswag/1573, hellaswag/1575, hellaswag/1577, hellaswag/1579, hellaswag/1584, hellaswag/1585, hellaswag/1603, hellaswag/1604, hellaswag/161, hellaswag/1610, hellaswag/1616, hellaswag/1617, hellaswag/1618, hellaswag/1626, hellaswag/163, hellaswag/1633, hellaswag/1640, hellaswag/1648, hellaswag/1650, hellaswag/1652, hellaswag/1654, hellaswag/1655, hellaswag/166, hellaswag/1661, hellaswag/1662, hellaswag/1663, hellaswag/1665, hellaswag/1666, hellaswag/1670, hellaswag/1672, hellaswag/1677, hellaswag/1683, hellaswag/1685, hellaswag/1687, hellaswag/1689, hellaswag/169, hellaswag/1698, hellaswag/1701, hellaswag/1703, hellaswag/1711, hellaswag/1713, hellaswag/1720, hellaswag/1721, hellaswag/1723, hellaswag/1726, hellaswag/1728, hellaswag/173, hellaswag/1736, hellaswag/1741, hellaswag/1743, hellaswag/1754, hellaswag/1755, hellaswag/1757, hellaswag/1762, hellaswag/1769, hellaswag/177, hellaswag/1774, hellaswag/1775, hellaswag/1779, hellaswag/1791, hellaswag/1794, hellaswag/1800, hellaswag/1806, hellaswag/1807, hellaswag/1812, hellaswag/1815, hellaswag/1819, hellaswag/1823, hellaswag/1825, hellaswag/1828, hellaswag/1829, hellaswag/183, hellaswag/1832, hellaswag/1835, hellaswag/1838, hellaswag/1840, hellaswag/1844, hellaswag/1845, hellaswag/1853, hellaswag/1854, hellaswag/186, hellaswag/1862, hellaswag/1867, hellaswag/1871, hellaswag/1872, hellaswag/1874, hellaswag/1875, hellaswag/1877, hellaswag/1878, hellaswag/1881, hellaswag/1888, hellaswag/1891, hellaswag/1892, hellaswag/1893, hellaswag/1895, hellaswag/190, hellaswag/1906, hellaswag/191, hellaswag/1910, hellaswag/1912, hellaswag/1916, hellaswag/1917, hellaswag/192, hellaswag/1936, hellaswag/1941, hellaswag/1947, hellaswag/1948, hellaswag/195, hellaswag/1959, hellaswag/1960, hellaswag/1961, hellaswag/1966, hellaswag/1970, hellaswag/1971, hellaswag/1976, hellaswag/1978, hellaswag/1980, hellaswag/1981, hellaswag/1984, hellaswag/1985, hellaswag/1987, hellaswag/1991, hellaswag/1997, hellaswag/20, hellaswag/2005, hellaswag/201, hellaswag/2011, hellaswag/2012, hellaswag/202, hellaswag/2027, hellaswag/2028, hellaswag/2031, hellaswag/2040, hellaswag/2046, hellaswag/2047, hellaswag/2055, hellaswag/2056, hellaswag/2058, hellaswag/2059, hellaswag/2063, hellaswag/2066, hellaswag/2074, hellaswag/2075, hellaswag/2083, hellaswag/2086, hellaswag/2087, hellaswag/2088, hellaswag/2093, hellaswag/2101, hellaswag/2104, hellaswag/2105, hellaswag/2106, hellaswag/2107, hellaswag/2108, hellaswag/2111, hellaswag/2112, hellaswag/2118, hellaswag/212, hellaswag/2126, hellaswag/2127, hellaswag/2128, hellaswag/2129, hellaswag/2134, hellaswag/2135, hellaswag/2138, hellaswag/2143, hellaswag/2146, hellaswag/2152, hellaswag/2162, hellaswag/2164, hellaswag/2165, hellaswag/2173, hellaswag/2176, hellaswag/2178, hellaswag/218, hellaswag/2182, hellaswag/2183, hellaswag/2187, hellaswag/2189, hellaswag/2195, hellaswag/2204, hellaswag/2206, hellaswag/2208, hellaswag/2209, hellaswag/221, hellaswag/2211, hellaswag/2216, hellaswag/2218, hellaswag/2231, hellaswag/2237, hellaswag/2238, hellaswag/224, hellaswag/2246, hellaswag/2249, hellaswag/2250, hellaswag/2251, hellaswag/2253, hellaswag/2255, hellaswag/2259, hellaswag/2262, hellaswag/2271, hellaswag/2285, hellaswag/2287, hellaswag/2292, hellaswag/2295, hellaswag/2299, hellaswag/230, hellaswag/2305, hellaswag/2306, hellaswag/2307, hellaswag/2308, hellaswag/2312, hellaswag/2316, hellaswag/232, hellaswag/2321, hellaswag/2323, hellaswag/2325, hellaswag/234, hellaswag/2340, hellaswag/2346, hellaswag/2347, hellaswag/2348, hellaswag/2349, hellaswag/235, hellaswag/2351, hellaswag/2353, hellaswag/2354, hellaswag/2355, hellaswag/2360, hellaswag/2368, hellaswag/2371, hellaswag/2376, hellaswag/2379, hellaswag/239, hellaswag/2390, hellaswag/2391, hellaswag/2393, hellaswag/2396, hellaswag/2397, hellaswag/240, hellaswag/2402, hellaswag/2404, hellaswag/2406, hellaswag/2409, hellaswag/2411, hellaswag/2419, hellaswag/2421, hellaswag/2424, hellaswag/2425, hellaswag/2427, hellaswag/2428, hellaswag/2429, hellaswag/2432, hellaswag/2436, hellaswag/2437, hellaswag/2439, hellaswag/2445, hellaswag/245, hellaswag/2451, hellaswag/2454, hellaswag/2466, hellaswag/2467, hellaswag/2481, hellaswag/2483, hellaswag/2495, hellaswag/2497, hellaswag/250, hellaswag/2505, hellaswag/251, hellaswag/2510, hellaswag/252, hellaswag/2521, hellaswag/2523, hellaswag/2525, hellaswag/2526, hellaswag/2535, hellaswag/2539, hellaswag/2546, hellaswag/2549, hellaswag/2552, hellaswag/2559, hellaswag/2560, hellaswag/2565, hellaswag/2566, hellaswag/2575, hellaswag/2577, hellaswag/2580, hellaswag/2583, hellaswag/2589, hellaswag/2596, hellaswag/2598, hellaswag/2599, hellaswag/2600, hellaswag/2602, hellaswag/2608, hellaswag/2609, hellaswag/261, hellaswag/2617, hellaswag/2618, hellaswag/2622, hellaswag/2624, hellaswag/2625, hellaswag/2627, hellaswag/2632, hellaswag/2634, hellaswag/2641, hellaswag/2646, hellaswag/2656, hellaswag/2661, hellaswag/2662, hellaswag/2664, hellaswag/267, hellaswag/2677, hellaswag/2681, hellaswag/2685, hellaswag/269, hellaswag/2690, hellaswag/2691, hellaswag/2699, hellaswag/270, hellaswag/2702, hellaswag/2712, hellaswag/2715, hellaswag/2719, hellaswag/2721, hellaswag/2722, hellaswag/2723, hellaswag/2733, hellaswag/2735, hellaswag/2737, hellaswag/2738, hellaswag/2742, hellaswag/2746, hellaswag/2749, hellaswag/2754, hellaswag/2759, hellaswag/2764, hellaswag/2765, hellaswag/2766, hellaswag/2767, hellaswag/2770, hellaswag/2771, hellaswag/2776, hellaswag/2777, hellaswag/278, hellaswag/2781, hellaswag/2783, hellaswag/2785, hellaswag/2787, hellaswag/2789, hellaswag/279, hellaswag/2792, hellaswag/2793, hellaswag/2794, hellaswag/2802, hellaswag/2805, hellaswag/2807, hellaswag/2808, hellaswag/2809, hellaswag/2813, hellaswag/2817, hellaswag/2819, hellaswag/2821, hellaswag/2822, hellaswag/2824, hellaswag/2829, hellaswag/2833, hellaswag/2838, hellaswag/2844, hellaswag/2848, hellaswag/285, hellaswag/2851, hellaswag/2852, hellaswag/2858, hellaswag/2859, hellaswag/286, hellaswag/2860, hellaswag/2866, hellaswag/2878, hellaswag/2886, hellaswag/2888, hellaswag/2889, hellaswag/289, hellaswag/2895, hellaswag/2898, hellaswag/2906, hellaswag/2908, hellaswag/291, hellaswag/2910, hellaswag/2912, hellaswag/2918, hellaswag/2921, hellaswag/2929, hellaswag/2934, hellaswag/2938, hellaswag/2942, hellaswag/2944, hellaswag/2951, hellaswag/2954, hellaswag/2956, hellaswag/296, hellaswag/2960, hellaswag/297, hellaswag/2976, hellaswag/2977, hellaswag/2981, hellaswag/2986, hellaswag/2990, hellaswag/30, hellaswag/3004, hellaswag/3005, hellaswag/3015, hellaswag/3019, hellaswag/302, hellaswag/3022, hellaswag/3023, hellaswag/3027, hellaswag/3029, hellaswag/3035, hellaswag/3037, hellaswag/3040, hellaswag/3041, hellaswag/3042, hellaswag/3047, hellaswag/3049, hellaswag/3059, hellaswag/306, hellaswag/3061, hellaswag/3062, hellaswag/3063, hellaswag/3075, hellaswag/3083, hellaswag/3086, hellaswag/3095, hellaswag/3099, hellaswag/3101, hellaswag/3104, hellaswag/3105, hellaswag/3106, hellaswag/3114, hellaswag/3117, hellaswag/3118, hellaswag/312, hellaswag/3122, hellaswag/3125, hellaswag/3126, hellaswag/3127, hellaswag/3133, hellaswag/3135, hellaswag/3136, hellaswag/3137, hellaswag/3138, hellaswag/3140, hellaswag/3144, hellaswag/3145, hellaswag/3150, hellaswag/3155, hellaswag/3156, hellaswag/3160, hellaswag/3164, hellaswag/3166, hellaswag/3168, hellaswag/3171, hellaswag/3176, hellaswag/3177, hellaswag/3179, hellaswag/3185, hellaswag/3188, hellaswag/3189, hellaswag/3190, hellaswag/3191, hellaswag/3195, hellaswag/3202, hellaswag/3204, hellaswag/3207, hellaswag/3209, hellaswag/3212, hellaswag/3225, hellaswag/3227, hellaswag/323, hellaswag/3231, hellaswag/3233, hellaswag/3234, hellaswag/3239, hellaswag/324, hellaswag/3242, hellaswag/3244, hellaswag/3247, hellaswag/3255, hellaswag/3266, hellaswag/327, hellaswag/3273, hellaswag/3277, hellaswag/3278, hellaswag/3284, hellaswag/3288, hellaswag/3289, hellaswag/3291, hellaswag/3298, hellaswag/3299, hellaswag/33, hellaswag/3310, hellaswag/3313, hellaswag/3314, hellaswag/3319, hellaswag/3328, hellaswag/3331, hellaswag/3338, hellaswag/3343, hellaswag/3346, hellaswag/3348, hellaswag/3349, hellaswag/335, hellaswag/3351, hellaswag/3357, hellaswag/336, hellaswag/3362, hellaswag/3367, hellaswag/3377, hellaswag/338, hellaswag/3382, hellaswag/3387, hellaswag/3390, hellaswag/3396, hellaswag/3398, hellaswag/3400, hellaswag/3401, hellaswag/3402, hellaswag/3413, hellaswag/342, hellaswag/3434, hellaswag/3435, hellaswag/3439, hellaswag/344, hellaswag/3441, hellaswag/3449, hellaswag/345, hellaswag/3453, hellaswag/3455, hellaswag/3457, hellaswag/3459, hellaswag/3461, hellaswag/3474, hellaswag/3477, hellaswag/3487, hellaswag/349, hellaswag/3498, hellaswag/3500, hellaswag/3504, hellaswag/3508, hellaswag/351, hellaswag/3513, hellaswag/3522, hellaswag/353, hellaswag/3533, hellaswag/354, hellaswag/3541, hellaswag/3542, hellaswag/3546, hellaswag/3553, hellaswag/3554, hellaswag/3555, hellaswag/3558, hellaswag/3562, hellaswag/3567, hellaswag/357, hellaswag/358, hellaswag/3592, hellaswag/3597, hellaswag/360, hellaswag/361, hellaswag/3611, hellaswag/3617, hellaswag/3627, hellaswag/3633, hellaswag/3634, hellaswag/3636, hellaswag/365, hellaswag/3659, hellaswag/3662, hellaswag/3666, hellaswag/3670, hellaswag/3675, hellaswag/368, hellaswag/3680, hellaswag/3682, hellaswag/3687, hellaswag/3689, hellaswag/3693, hellaswag/3694, hellaswag/3702, hellaswag/3703, hellaswag/3707, hellaswag/371, hellaswag/3719, hellaswag/3721, hellaswag/3723, hellaswag/3725, hellaswag/3727, hellaswag/3740, hellaswag/3742, hellaswag/3746, hellaswag/3747, hellaswag/375, hellaswag/3752, hellaswag/3756, hellaswag/3758, hellaswag/3764, hellaswag/3766, hellaswag/3767, hellaswag/3776, hellaswag/3780, hellaswag/3781, hellaswag/380, hellaswag/3801, hellaswag/3802, hellaswag/3810, hellaswag/3814, hellaswag/3818, hellaswag/3819, hellaswag/3826, hellaswag/3830, hellaswag/3835, hellaswag/384, hellaswag/3842, hellaswag/3843, hellaswag/3844, hellaswag/3861, hellaswag/3867, hellaswag/3876, hellaswag/3877, hellaswag/389, hellaswag/3890, hellaswag/3893, hellaswag/3895, hellaswag/3897, hellaswag/3898, hellaswag/3904, hellaswag/3908, hellaswag/3914, hellaswag/3915, hellaswag/392, hellaswag/3922, hellaswag/3926, hellaswag/393, hellaswag/3931, hellaswag/3934, hellaswag/3939, hellaswag/3946, hellaswag/395, hellaswag/3953, hellaswag/3959, hellaswag/3960, hellaswag/3965, hellaswag/3966, hellaswag/3969, hellaswag/3979, hellaswag/3980, hellaswag/3981, hellaswag/3984, hellaswag/3985, hellaswag/3989, hellaswag/3990, hellaswag/3994, hellaswag/3995, hellaswag/400, hellaswag/4002, hellaswag/4007, hellaswag/401, hellaswag/4017, hellaswag/4021, hellaswag/4026, hellaswag/4028, hellaswag/4038, hellaswag/4041, hellaswag/4045, hellaswag/4046, hellaswag/4053, hellaswag/4054, hellaswag/4056, hellaswag/4077, hellaswag/4080, hellaswag/4081, hellaswag/4082, hellaswag/4101, hellaswag/4112, hellaswag/4119, hellaswag/4120, hellaswag/4146, hellaswag/415, hellaswag/4157, hellaswag/4162, hellaswag/4165, hellaswag/4167, hellaswag/4169, hellaswag/4173, hellaswag/4178, hellaswag/4180, hellaswag/4182, hellaswag/4183, hellaswag/4184, hellaswag/419, hellaswag/4192, hellaswag/4193, hellaswag/4195, hellaswag/4202, hellaswag/4206, hellaswag/421, hellaswag/4211, hellaswag/4215, hellaswag/4218, hellaswag/4219, hellaswag/4222, hellaswag/4224, hellaswag/4234, hellaswag/4235, hellaswag/4236, hellaswag/4237, hellaswag/4239, hellaswag/4240, hellaswag/425, hellaswag/4250, hellaswag/4270, hellaswag/4276, hellaswag/4278, hellaswag/4285, hellaswag/4289, hellaswag/4291, hellaswag/4292, hellaswag/4305, hellaswag/4306, hellaswag/4308, hellaswag/4335, hellaswag/4337, hellaswag/434, hellaswag/4340, hellaswag/4345, hellaswag/436, hellaswag/4360, hellaswag/4361, hellaswag/4374, hellaswag/4375, hellaswag/4381, hellaswag/4386, hellaswag/4388, hellaswag/4390, hellaswag/440, hellaswag/4413, hellaswag/4417, hellaswag/4420, hellaswag/4426, hellaswag/4436, hellaswag/4437, hellaswag/4441, hellaswag/445, hellaswag/4451, hellaswag/4459, hellaswag/4466, hellaswag/4482, hellaswag/4488, hellaswag/4492, hellaswag/4493, hellaswag/4495, hellaswag/4497, hellaswag/4506, hellaswag/4509, hellaswag/451, hellaswag/4513, hellaswag/4516, hellaswag/452, hellaswag/4520, hellaswag/4524, hellaswag/454, hellaswag/4540, hellaswag/4542, hellaswag/4544, hellaswag/4549, hellaswag/455, hellaswag/4550, hellaswag/4551, hellaswag/4555, hellaswag/4557, hellaswag/4558, hellaswag/4563, hellaswag/4569, hellaswag/4575, hellaswag/4579, hellaswag/4591, hellaswag/4592, hellaswag/4593, hellaswag/4600, hellaswag/4603, hellaswag/4626, hellaswag/4631, hellaswag/4641, hellaswag/4642, hellaswag/4645, hellaswag/465, hellaswag/4654, hellaswag/4656, hellaswag/466, hellaswag/4660, hellaswag/4665, hellaswag/468, hellaswag/4683, hellaswag/4688, hellaswag/4690, hellaswag/47, hellaswag/4700, hellaswag/4706, hellaswag/4710, hellaswag/4712, hellaswag/4713, hellaswag/4714, hellaswag/4718, hellaswag/4727, hellaswag/4730, hellaswag/4734, hellaswag/4735, hellaswag/4736, hellaswag/4737, hellaswag/474, hellaswag/4740, hellaswag/4744, hellaswag/4749, hellaswag/475, hellaswag/4750, hellaswag/4752, hellaswag/4753, hellaswag/4755, hellaswag/4756, hellaswag/4764, hellaswag/4767, hellaswag/4771, hellaswag/4785, hellaswag/4788, hellaswag/4790, hellaswag/4797, hellaswag/48, hellaswag/4802, hellaswag/4804, hellaswag/482, hellaswag/4823, hellaswag/4827, hellaswag/4836, hellaswag/4841, hellaswag/4847, hellaswag/4848, hellaswag/4852, hellaswag/4854, hellaswag/4862, hellaswag/4869, hellaswag/487, hellaswag/4884, hellaswag/4885, hellaswag/4887, hellaswag/4891, hellaswag/4892, hellaswag/4894, hellaswag/4897, hellaswag/4899, hellaswag/4905, hellaswag/4906, hellaswag/4913, hellaswag/4923, hellaswag/4936, hellaswag/4937, hellaswag/4949, hellaswag/4958, hellaswag/496, hellaswag/4963, hellaswag/4976, hellaswag/498, hellaswag/4982, hellaswag/50, hellaswag/5006, hellaswag/5008, hellaswag/501, hellaswag/5011, hellaswag/5021, hellaswag/5023, hellaswag/5024, hellaswag/5034, hellaswag/5038, hellaswag/505, hellaswag/5055, hellaswag/5065, hellaswag/5068, hellaswag/5070, hellaswag/5073, hellaswag/5078, hellaswag/5089, hellaswag/5090, hellaswag/5096, hellaswag/510, hellaswag/5100, hellaswag/5105, hellaswag/5109, hellaswag/5115, hellaswag/5117, hellaswag/5124, hellaswag/5125, hellaswag/5126, hellaswag/5127, hellaswag/513, hellaswag/5131, hellaswag/5132, hellaswag/514, hellaswag/515, hellaswag/5151, hellaswag/5153, hellaswag/5154, hellaswag/5156, hellaswag/5169, hellaswag/517, hellaswag/5175, hellaswag/5180, hellaswag/5197, hellaswag/5198, hellaswag/5209, hellaswag/5210, hellaswag/5219, hellaswag/5232, hellaswag/5237, hellaswag/5248, hellaswag/5256, hellaswag/5259, hellaswag/5261, hellaswag/5264, hellaswag/5271, hellaswag/5279, hellaswag/5282, hellaswag/5283, hellaswag/5291, hellaswag/5292, hellaswag/5293, hellaswag/5294, hellaswag/53, hellaswag/530, hellaswag/5300, hellaswag/5301, hellaswag/5310, hellaswag/5317, hellaswag/5319, hellaswag/5323, hellaswag/5326, hellaswag/5330, hellaswag/5338, hellaswag/534, hellaswag/5345, hellaswag/5352, hellaswag/536, hellaswag/5362, hellaswag/5377, hellaswag/5383, hellaswag/5397, hellaswag/54, hellaswag/5403, hellaswag/5404, hellaswag/5407, hellaswag/5412, hellaswag/5414, hellaswag/5424, hellaswag/5427, hellaswag/5434, hellaswag/544, hellaswag/5446, hellaswag/5447, hellaswag/545, hellaswag/546, hellaswag/5461, hellaswag/5464, hellaswag/5469, hellaswag/5477, hellaswag/5478, hellaswag/5481, hellaswag/5482, hellaswag/549, hellaswag/5493, hellaswag/5505, hellaswag/5513, hellaswag/5514, hellaswag/5524, hellaswag/5529, hellaswag/5544, hellaswag/5552, hellaswag/5554, hellaswag/5568, hellaswag/5572, hellaswag/558, hellaswag/5580, hellaswag/5591, hellaswag/5608, hellaswag/561, hellaswag/5611, hellaswag/5617, hellaswag/5619, hellaswag/562, hellaswag/5622, hellaswag/5629, hellaswag/5635, hellaswag/564, hellaswag/5640, hellaswag/5646, hellaswag/5661, hellaswag/5670, hellaswag/5690, hellaswag/5692, hellaswag/5707, hellaswag/5708, hellaswag/5714, hellaswag/5724, hellaswag/5727, hellaswag/5729, hellaswag/574, hellaswag/5742, hellaswag/5743, hellaswag/5744, hellaswag/5749, hellaswag/5759, hellaswag/5762, hellaswag/5769, hellaswag/5773, hellaswag/578, hellaswag/5780, hellaswag/579, hellaswag/5794, hellaswag/5797, hellaswag/58, hellaswag/5805, hellaswag/581, hellaswag/5825, hellaswag/583, hellaswag/5832, hellaswag/5833, hellaswag/5836, hellaswag/5855, hellaswag/5863, hellaswag/5867, hellaswag/5871, hellaswag/5872, hellaswag/5874, hellaswag/5877, hellaswag/5880, hellaswag/5883, hellaswag/5896, hellaswag/5897, hellaswag/5898, hellaswag/59, hellaswag/5900, hellaswag/591, hellaswag/5916, hellaswag/5925, hellaswag/5927, hellaswag/5935, hellaswag/5944, hellaswag/5945, hellaswag/5954, hellaswag/5959, hellaswag/5961, hellaswag/5966, hellaswag/5970, hellaswag/5974, hellaswag/5979, hellaswag/5982, hellaswag/5995, hellaswag/5996, hellaswag/6, hellaswag/60, hellaswag/6017, hellaswag/6019, hellaswag/6025, hellaswag/6027, hellaswag/6030, hellaswag/6031, hellaswag/6035, hellaswag/6038, hellaswag/6039, hellaswag/6041, hellaswag/605, hellaswag/6051, hellaswag/6053, hellaswag/6060, hellaswag/6063, hellaswag/6071, hellaswag/6082, hellaswag/6083, hellaswag/609, hellaswag/610, hellaswag/6123, hellaswag/6127, hellaswag/6128, hellaswag/6130, hellaswag/6135, hellaswag/6138, hellaswag/614, hellaswag/6141, hellaswag/6150, hellaswag/6152, hellaswag/6155, hellaswag/6157, hellaswag/616, hellaswag/6161, hellaswag/6164, hellaswag/6168, hellaswag/617, hellaswag/6171, hellaswag/6172, hellaswag/6175, hellaswag/6177, hellaswag/6200, hellaswag/6204, hellaswag/6206, hellaswag/6210, hellaswag/6215, hellaswag/6218, hellaswag/6222, hellaswag/6226, hellaswag/6228, hellaswag/6230, hellaswag/6235, hellaswag/6250, hellaswag/6251, hellaswag/6253, hellaswag/6262, hellaswag/6271, hellaswag/6273, hellaswag/6275, hellaswag/6283, hellaswag/6285, hellaswag/6286, hellaswag/6292, hellaswag/630, hellaswag/6307, hellaswag/6316, hellaswag/6317, hellaswag/6320, hellaswag/6323, hellaswag/6329, hellaswag/6330, hellaswag/6332, hellaswag/6337, hellaswag/6345, hellaswag/6358, hellaswag/6363, hellaswag/6372, hellaswag/6381, hellaswag/6383, hellaswag/6385, hellaswag/639, hellaswag/64, hellaswag/6400, hellaswag/6402, hellaswag/6406, hellaswag/6413, hellaswag/642, hellaswag/6422, hellaswag/6430, hellaswag/6432, hellaswag/6443, hellaswag/6444, hellaswag/6448, hellaswag/6449, hellaswag/645, hellaswag/6450, hellaswag/6456, hellaswag/648, hellaswag/6480, hellaswag/6482, hellaswag/6483, hellaswag/649, hellaswag/6490, hellaswag/6493, hellaswag/6498, hellaswag/6501, hellaswag/651, hellaswag/6513, hellaswag/6514, hellaswag/6520, hellaswag/6521, hellaswag/6530, hellaswag/6533, hellaswag/6534, hellaswag/6547, hellaswag/6565, hellaswag/6567, hellaswag/6568, hellaswag/657, hellaswag/6578, hellaswag/6588, hellaswag/6592, hellaswag/6593, hellaswag/6600, hellaswag/6602, hellaswag/6605, hellaswag/661, hellaswag/6626, hellaswag/6634, hellaswag/6635, hellaswag/6637, hellaswag/6644, hellaswag/6646, hellaswag/6649, hellaswag/665, hellaswag/6654, hellaswag/6668, hellaswag/6675, hellaswag/668, hellaswag/6688, hellaswag/669, hellaswag/6706, hellaswag/671, hellaswag/6710, hellaswag/6714, hellaswag/6718, hellaswag/672, hellaswag/6722, hellaswag/6725, hellaswag/6732, hellaswag/6737, hellaswag/6739, hellaswag/674, hellaswag/6741, hellaswag/6743, hellaswag/6750, hellaswag/6756, hellaswag/6757, hellaswag/6758, hellaswag/6761, hellaswag/6762, hellaswag/6763, hellaswag/6766, hellaswag/6767, hellaswag/6768, hellaswag/677, hellaswag/6771, hellaswag/6774, hellaswag/6777, hellaswag/6780, hellaswag/6784, hellaswag/6786, hellaswag/6790, hellaswag/6798, hellaswag/6799, hellaswag/68, hellaswag/6807, hellaswag/6811, hellaswag/6812, hellaswag/6815, hellaswag/6816, hellaswag/6819, hellaswag/6825, hellaswag/6827, hellaswag/6831, hellaswag/6833, hellaswag/6834, hellaswag/6839, hellaswag/684, hellaswag/6842, hellaswag/6843, hellaswag/6845, hellaswag/6848, hellaswag/6853, hellaswag/6877, hellaswag/6878, hellaswag/6883, hellaswag/6887, hellaswag/6888, hellaswag/6891, hellaswag/6893, hellaswag/6900, hellaswag/6904, hellaswag/6914, hellaswag/6918, hellaswag/692, hellaswag/6936, hellaswag/6937, hellaswag/6939, hellaswag/6944, hellaswag/6945, hellaswag/6951, hellaswag/6963, hellaswag/6969, hellaswag/6970, hellaswag/698, hellaswag/6980, hellaswag/6982, hellaswag/6989, hellaswag/6991, hellaswag/6996, hellaswag/7, hellaswag/70, hellaswag/7000, hellaswag/7002, hellaswag/7004, hellaswag/7006, hellaswag/702, hellaswag/7022, hellaswag/7023, hellaswag/7028, hellaswag/7029, hellaswag/7031, hellaswag/7032, hellaswag/7044, hellaswag/7048, hellaswag/705, hellaswag/7057, hellaswag/706, hellaswag/7060, hellaswag/7070, hellaswag/7077, hellaswag/7079, hellaswag/708, hellaswag/7085, hellaswag/709, hellaswag/71, hellaswag/7104, hellaswag/7108, hellaswag/7114, hellaswag/712, hellaswag/7124, hellaswag/7129, hellaswag/7130, hellaswag/7134, hellaswag/7142, hellaswag/7148, hellaswag/719, hellaswag/7191, hellaswag/7193, hellaswag/7196, hellaswag/7199, hellaswag/7203, hellaswag/7205, hellaswag/7211, hellaswag/7214, hellaswag/7222, hellaswag/7226, hellaswag/7227, hellaswag/724, hellaswag/7245, hellaswag/7246, hellaswag/7256, hellaswag/726, hellaswag/7260, hellaswag/7269, hellaswag/727, hellaswag/7271, hellaswag/7274, hellaswag/7280, hellaswag/7283, hellaswag/7285, hellaswag/7295, hellaswag/730, hellaswag/7301, hellaswag/7302, hellaswag/7316, hellaswag/732, hellaswag/7330, hellaswag/7336, hellaswag/7339, hellaswag/7344, hellaswag/7348, hellaswag/7354, hellaswag/7355, hellaswag/7364, hellaswag/7366, hellaswag/7384, hellaswag/7389, hellaswag/7395, hellaswag/740, hellaswag/7403, hellaswag/7405, hellaswag/7408, hellaswag/7414, hellaswag/7415, hellaswag/7417, hellaswag/7418, hellaswag/7421, hellaswag/7422, hellaswag/7424, hellaswag/7432, hellaswag/7436, hellaswag/744, hellaswag/7451, hellaswag/7459, hellaswag/7463, hellaswag/7465, hellaswag/747, hellaswag/7474, hellaswag/7479, hellaswag/7483, hellaswag/7497, hellaswag/750, hellaswag/7501, hellaswag/7503, hellaswag/7506, hellaswag/7510, hellaswag/7512, hellaswag/7517, hellaswag/752, hellaswag/7521, hellaswag/7523, hellaswag/7527, hellaswag/7529, hellaswag/7533, hellaswag/754, hellaswag/7548, hellaswag/755, hellaswag/7554, hellaswag/7556, hellaswag/756, hellaswag/7563, hellaswag/7564, hellaswag/7568, hellaswag/7574, hellaswag/7577, hellaswag/7583, hellaswag/7584, hellaswag/759, hellaswag/7591, hellaswag/7594, hellaswag/7595, hellaswag/7598, hellaswag/7604, hellaswag/7612, hellaswag/7624, hellaswag/7627, hellaswag/7628, hellaswag/7632, hellaswag/7634, hellaswag/7649, hellaswag/7651, hellaswag/7653, hellaswag/7654, hellaswag/766, hellaswag/7660, hellaswag/7661, hellaswag/7662, hellaswag/7674, hellaswag/7696, hellaswag/7702, hellaswag/7712, hellaswag/7721, hellaswag/774, hellaswag/7756, hellaswag/7758, hellaswag/7759, hellaswag/7760, hellaswag/7766, hellaswag/7767, hellaswag/7774, hellaswag/7781, hellaswag/7810, hellaswag/7813, hellaswag/7818, hellaswag/7822, hellaswag/7823, hellaswag/7825, hellaswag/7827, hellaswag/783, hellaswag/7842, hellaswag/785, hellaswag/7860, hellaswag/7862, hellaswag/7864, hellaswag/7868, hellaswag/7875, hellaswag/7876, hellaswag/788, hellaswag/7884, hellaswag/7887, hellaswag/789, hellaswag/7903, hellaswag/7910, hellaswag/7911, hellaswag/7912, hellaswag/7915, hellaswag/7916, hellaswag/7917, hellaswag/7923, hellaswag/7931, hellaswag/7936, hellaswag/7943, hellaswag/7946, hellaswag/7947, hellaswag/7953, hellaswag/7960, hellaswag/7962, hellaswag/7971, hellaswag/798, hellaswag/7982, hellaswag/7989, hellaswag/80, hellaswag/800, hellaswag/8020, hellaswag/8026, hellaswag/8027, hellaswag/803, hellaswag/8038, hellaswag/8041, hellaswag/8046, hellaswag/8048, hellaswag/8050, hellaswag/8052, hellaswag/8059, hellaswag/806, hellaswag/8068, hellaswag/8075, hellaswag/8077, hellaswag/8090, hellaswag/8093, hellaswag/8094, hellaswag/8095, hellaswag/810, hellaswag/8111, hellaswag/8127, hellaswag/8132, hellaswag/8133, hellaswag/8134, hellaswag/8137, hellaswag/814, hellaswag/8145, hellaswag/8149, hellaswag/8153, hellaswag/8161, hellaswag/8166, hellaswag/8172, hellaswag/8174, hellaswag/818, hellaswag/8181, hellaswag/8194, hellaswag/8195, hellaswag/8196, hellaswag/820, hellaswag/8202, hellaswag/8205, hellaswag/8209, hellaswag/821, hellaswag/8218, hellaswag/822, hellaswag/8220, hellaswag/8221, hellaswag/8225, hellaswag/8226, hellaswag/8227, hellaswag/8230, hellaswag/8241, hellaswag/8246, hellaswag/8251, hellaswag/826, hellaswag/8268, hellaswag/8271, hellaswag/828, hellaswag/8281, hellaswag/8290, hellaswag/8291, hellaswag/8293, hellaswag/8294, hellaswag/8298, hellaswag/8309, hellaswag/8315, hellaswag/8329, hellaswag/833, hellaswag/8353, hellaswag/8354, hellaswag/8357, hellaswag/8359, hellaswag/836, hellaswag/8365, hellaswag/8370, hellaswag/8372, hellaswag/8376, hellaswag/8379, hellaswag/8382, hellaswag/8389, hellaswag/839, hellaswag/8391, hellaswag/8393, hellaswag/84, hellaswag/840, hellaswag/8408, hellaswag/8411, hellaswag/8419, hellaswag/8437, hellaswag/844, hellaswag/845, hellaswag/8450, hellaswag/8451, hellaswag/8452, hellaswag/8454, hellaswag/8456, hellaswag/8459, hellaswag/8462, hellaswag/8465, hellaswag/8466, hellaswag/8473, hellaswag/8476, hellaswag/8487, hellaswag/8490, hellaswag/8491, hellaswag/8500, hellaswag/8501, hellaswag/8502, hellaswag/8510, hellaswag/8511, hellaswag/8512, hellaswag/8514, hellaswag/8517, hellaswag/8533, hellaswag/854, hellaswag/8540, hellaswag/8551, hellaswag/8552, hellaswag/8561, hellaswag/8564, hellaswag/8568, hellaswag/857, hellaswag/8575, hellaswag/8582, hellaswag/8587, hellaswag/859, hellaswag/8593, hellaswag/8594, hellaswag/86, hellaswag/8607, hellaswag/8611, hellaswag/8619, hellaswag/862, hellaswag/8620, hellaswag/8621, hellaswag/8623, hellaswag/8631, hellaswag/8634, hellaswag/8636, hellaswag/8642, hellaswag/8643, hellaswag/8645, hellaswag/8647, hellaswag/8663, hellaswag/8670, hellaswag/8672, hellaswag/8682, hellaswag/8689, hellaswag/869, hellaswag/8692, hellaswag/8693, hellaswag/8694, hellaswag/8698, hellaswag/8702, hellaswag/8703, hellaswag/8704, hellaswag/8718, hellaswag/872, hellaswag/8727, hellaswag/8728, hellaswag/8734, hellaswag/8745, hellaswag/875, hellaswag/8756, hellaswag/8762, hellaswag/8766, hellaswag/8767, hellaswag/8780, hellaswag/8782, hellaswag/8789, hellaswag/8797, hellaswag/8805, hellaswag/8808, hellaswag/8809, hellaswag/8817, hellaswag/8826, hellaswag/8829, hellaswag/8830, hellaswag/8838, hellaswag/884, hellaswag/8840, hellaswag/8860, hellaswag/8864, hellaswag/8869, hellaswag/8873, hellaswag/8888, hellaswag/8894, hellaswag/8895, hellaswag/8896, hellaswag/8898, hellaswag/8899, hellaswag/8901, hellaswag/8902, hellaswag/8905, hellaswag/8919, hellaswag/892, hellaswag/8928, hellaswag/8930, hellaswag/8934, hellaswag/8942, hellaswag/8947, hellaswag/895, hellaswag/8950, hellaswag/8952, hellaswag/8957, hellaswag/8971, hellaswag/8978, hellaswag/8980, hellaswag/8981, hellaswag/8984, hellaswag/899, hellaswag/8990, hellaswag/8995, hellaswag/9, hellaswag/90, hellaswag/900, hellaswag/9006, hellaswag/9022, hellaswag/9025, hellaswag/9033, hellaswag/9034, hellaswag/9035, hellaswag/904, hellaswag/9040, hellaswag/9044, hellaswag/9046, hellaswag/9057, hellaswag/9058, hellaswag/9060, hellaswag/9067, hellaswag/9069, hellaswag/9071, hellaswag/9072, hellaswag/9091, hellaswag/9112, hellaswag/9128, hellaswag/913, hellaswag/9130, hellaswag/9132, hellaswag/9134, hellaswag/9136, hellaswag/914, hellaswag/9144, hellaswag/915, hellaswag/9151, hellaswag/9157, hellaswag/9159, hellaswag/9160, hellaswag/9166, hellaswag/9170, hellaswag/9173, hellaswag/9185, hellaswag/9187, hellaswag/9198, hellaswag/9208, hellaswag/921, hellaswag/9216, hellaswag/9220, hellaswag/9223, hellaswag/9226, hellaswag/923, hellaswag/9249, hellaswag/9252, hellaswag/927, hellaswag/9271, hellaswag/9278, hellaswag/9279, hellaswag/928, hellaswag/9280, hellaswag/9281, hellaswag/9284, hellaswag/9286, hellaswag/932, hellaswag/9326, hellaswag/9327, hellaswag/9329, hellaswag/9332, hellaswag/9339, hellaswag/9340, hellaswag/9352, hellaswag/9354, hellaswag/9359, hellaswag/9361, hellaswag/9365, hellaswag/9367, hellaswag/9368, hellaswag/9370, hellaswag/9371, hellaswag/9377, hellaswag/9378, hellaswag/938, hellaswag/939, hellaswag/9391, hellaswag/9395, hellaswag/9400, hellaswag/9401, hellaswag/9402, hellaswag/9405, hellaswag/9407, hellaswag/941, hellaswag/9410, hellaswag/9411, hellaswag/9412, hellaswag/9413, hellaswag/9416, hellaswag/942, hellaswag/9425, hellaswag/9432, hellaswag/9433, hellaswag/9437, hellaswag/9441, hellaswag/945, hellaswag/9451, hellaswag/9478, hellaswag/948, hellaswag/9481, hellaswag/9495, hellaswag/9498, hellaswag/9502, hellaswag/9508, hellaswag/9512, hellaswag/9517, hellaswag/952, hellaswag/9520, hellaswag/9524, hellaswag/9525, hellaswag/9540, hellaswag/9541, hellaswag/9543, hellaswag/9544, hellaswag/9551, hellaswag/9554, hellaswag/9576, hellaswag/9583, hellaswag/9597, hellaswag/96, hellaswag/9607, hellaswag/9615, hellaswag/962, hellaswag/9629, hellaswag/9630, hellaswag/9637, hellaswag/9638, hellaswag/9644, hellaswag/9646, hellaswag/9647, hellaswag/965, hellaswag/9653, hellaswag/9655, hellaswag/9657, hellaswag/9666, hellaswag/9668, hellaswag/9669, hellaswag/967, hellaswag/9673, hellaswag/968, hellaswag/9684, hellaswag/9695, hellaswag/9696, hellaswag/9702, hellaswag/9709, hellaswag/9711, hellaswag/9716, hellaswag/972, hellaswag/9730, hellaswag/9740, hellaswag/9743, hellaswag/9747, hellaswag/9749, hellaswag/975, hellaswag/9754, hellaswag/9762, hellaswag/9770, hellaswag/9771, hellaswag/9778, hellaswag/9779, hellaswag/978, hellaswag/9785, hellaswag/9788, hellaswag/979, hellaswag/9797, hellaswag/9798, hellaswag/9799, hellaswag/98, hellaswag/980, hellaswag/9809, hellaswag/981, hellaswag/9823, hellaswag/9833, hellaswag/9834, hellaswag/9837, hellaswag/9844, hellaswag/9846, hellaswag/9847, hellaswag/9849, hellaswag/9853, hellaswag/987, hellaswag/9873, hellaswag/9878, hellaswag/9879, hellaswag/9886, hellaswag/9887, hellaswag/9889, hellaswag/9893, hellaswag/9898, hellaswag/9899, hellaswag/9902, hellaswag/9904, hellaswag/9911, hellaswag/9912, hellaswag/9913, hellaswag/992, hellaswag/9921, hellaswag/9922, hellaswag/9931, hellaswag/9938, hellaswag/9945, hellaswag/9946, hellaswag/9956, hellaswag/9966, hellaswag/997, hellaswag/9973, hellaswag/9976, hellaswag/9985, hellaswag/9992

Problems solved by 1 model only

example_link model min_pass1_of_model
hellaswag/3142 seed27712 0.703
hellaswag/2022 seed27712 0.703
hellaswag/3657 seed27712 0.703
hellaswag/3628 seed27712 0.703
hellaswag/2245 seed27712 0.703
hellaswag/319 seed27712 0.703
hellaswag/256 seed27712 0.703
hellaswag/8443 seed27712 0.703
hellaswag/115 seed27712 0.703
hellaswag/5084 seed27712 0.703
hellaswag/6623 seed27712 0.703
hellaswag/4116 seed27712 0.703
hellaswag/385 seed27712 0.703
hellaswag/620 seed27712 0.703
hellaswag/6296 seed27712 0.703
hellaswag/6748 seed27712 0.703
hellaswag/6101 seed27712 0.703
hellaswag/1580 seed27712 0.703
hellaswag/1381 seed27712 0.703
hellaswag/1676 seed27712 0.703
hellaswag/1642 seed27712 0.703
hellaswag/1967 seed27712 0.703
hellaswag/9680 seed27712 0.703
hellaswag/9787 seed27712 0.703
hellaswag/9587 seed27712 0.703
hellaswag/7137 seed27712 0.703
hellaswag/7009 seed27712 0.703
hellaswag/4776 seed27712 0.703
hellaswag/4723 seed27712 0.703
hellaswag/5254 seed27712 0.703
hellaswag/502 seed27712 0.703
hellaswag/506 seed27712 0.703
hellaswag/5456 seed27712 0.703
hellaswag/7447 seed27712 0.703
hellaswag/7560 seed27712 0.703
hellaswag/7443 seed27712 0.703
hellaswag/7439 seed27712 0.703
hellaswag/9997 seed149 0.702
hellaswag/3741 seed149 0.702
hellaswag/7590 seed149 0.702
hellaswag/1050 seed149 0.702
hellaswag/7149 seed149 0.702
hellaswag/7498 seed149 0.702
hellaswag/4176 seed149 0.702
hellaswag/4613 seed149 0.702
hellaswag/1047 seed149 0.702
hellaswag/1049 seed149 0.702
hellaswag/9827 seed149 0.702
hellaswag/1008 seed149 0.702
hellaswag/807 seed149 0.702
hellaswag/2350 seed149 0.702
hellaswag/8164 seed149 0.702
hellaswag/7433 seed149 0.702
hellaswag/6886 seed149 0.702
hellaswag/4387 seed149 0.702
hellaswag/7670 seed149 0.702
hellaswag/7638 seed149 0.702
hellaswag/9336 seed149 0.702
hellaswag/8881 seed149 0.702
hellaswag/6197 seed149 0.702
hellaswag/6364 seed149 0.702
hellaswag/982 seed149 0.702
hellaswag/1866 seed149 0.702
hellaswag/9482 seed149 0.702
hellaswag/898 seed149 0.702
hellaswag/5376 seed149 0.702
hellaswag/4944 seed149 0.702
hellaswag/768 seed149 0.702
hellaswag/6086 seed149 0.702
hellaswag/6223 seed149 0.702
hellaswag/2633 seed6529 0.702
hellaswag/2594 seed6529 0.702
hellaswag/204 seed6529 0.702
hellaswag/1472 seed6529 0.702
hellaswag/1454 seed6529 0.702
hellaswag/9066 seed6529 0.702
hellaswag/198 seed6529 0.702
hellaswag/1582 seed6529 0.702
hellaswag/6458 seed6529 0.702
hellaswag/6005 seed6529 0.702
hellaswag/963 seed6529 0.702
hellaswag/1473 seed6529 0.702
hellaswag/5557 seed6529 0.702
hellaswag/7460 seed6529 0.702
hellaswag/3706 seed6529 0.702
hellaswag/2140 seed6529 0.702
hellaswag/2435 seed6529 0.702
hellaswag/2933 seed6529 0.702
hellaswag/3356 seed6529 0.702
hellaswag/8397 seed6529 0.702
hellaswag/8171 seed6529 0.702
hellaswag/4369 seed6529 0.702
hellaswag/5273 seed6529 0.702
hellaswag/3987 seed6529 0.702
hellaswag/1798 seed6529 0.702
hellaswag/6024 seed6529 0.702
hellaswag/6042 seed6529 0.702
hellaswag/5462 seed6529 0.702
hellaswag/5808 seed6529 0.702
hellaswag/7731 seed6529 0.702
hellaswag/7188 seed6529 0.702
hellaswag/543 seed6529 0.702
hellaswag/4693 seed6529 0.702
hellaswag/431 seed16385 0.701
hellaswag/6369 seed16385 0.701
hellaswag/8470 seed16385 0.701
hellaswag/8394 seed16385 0.701
hellaswag/83 seed16385 0.701
hellaswag/8186 seed16385 0.701
hellaswag/8228 seed16385 0.701
hellaswag/7950 seed16385 0.701
hellaswag/2561 seed16385 0.701
hellaswag/2761 seed16385 0.701
hellaswag/3302 seed16385 0.701
hellaswag/2881 seed16385 0.701
hellaswag/2891 seed16385 0.701
hellaswag/6309 seed16385 0.701
hellaswag/977 seed16385 0.701
hellaswag/9672 seed16385 0.701
hellaswag/9707 seed16385 0.701
hellaswag/1938 seed16385 0.701
hellaswag/5683 seed16385 0.701
hellaswag/3836 seed16385 0.701
hellaswag/4585 seed16385 0.701
hellaswag/3971 seed16385 0.701
hellaswag/4545 seed16385 0.701
hellaswag/449 seed16385 0.701
hellaswag/7265 seed16385 0.701
hellaswag/7807 seed16385 0.701
hellaswag/402 seed16385 0.701
hellaswag/5394 seed16385 0.701
hellaswag/5026 seed16385 0.701
hellaswag/3556 seed16385 0.701
hellaswag/3714 seed16385 0.701
hellaswag/10016 seed14309 0.701
hellaswag/986 seed14309 0.701
hellaswag/1057 seed14309 0.701
hellaswag/1836 seed14309 0.701
hellaswag/1646 seed14309 0.701
hellaswag/1761 seed14309 0.701
hellaswag/1712 seed14309 0.701
hellaswag/8674 seed14309 0.701
hellaswag/398 seed14309 0.701
hellaswag/5517 seed14309 0.701
hellaswag/529 seed14309 0.701
hellaswag/486 seed14309 0.701
hellaswag/7253 seed14309 0.701
hellaswag/6510 seed14309 0.701
hellaswag/6724 seed14309 0.701
hellaswag/6052 seed14309 0.701
hellaswag/9719 seed14309 0.701
hellaswag/9765 seed14309 0.701
hellaswag/1238 seed14309 0.701
hellaswag/1846 seed14309 0.701
hellaswag/3421 seed14309 0.701
hellaswag/8 seed14309 0.701
hellaswag/3292 seed14309 0.701
hellaswag/8569 seed14309 0.701
hellaswag/8126 seed14309 0.701
hellaswag/2704 seed14309 0.701
hellaswag/2352 seed14309 0.701
hellaswag/2971 seed14309 0.701
hellaswag/242 seed14309 0.701
hellaswag/2240 seed14309 0.701
hellaswag/7164 seed14309 0.701
hellaswag/4467 seed14309 0.701
hellaswag/463 seed14309 0.701
hellaswag/5697 seed14309 0.701
hellaswag/7419 seed14309 0.701
hellaswag/9503 seed14309 0.701
hellaswag/2950 seed24734 0.700
hellaswag/3279 seed24734 0.700
hellaswag/3065 seed24734 0.700
hellaswag/2095 seed24734 0.700
hellaswag/2037 seed24734 0.700
hellaswag/2489 seed24734 0.700
hellaswag/2190 seed24734 0.700
hellaswag/7942 seed24734 0.700
hellaswag/793 seed24734 0.700
hellaswag/8214 seed24734 0.700
hellaswag/8082 seed24734 0.700
hellaswag/2769 seed24734 0.700
hellaswag/2711 seed24734 0.700
hellaswag/2814 seed24734 0.700
hellaswag/2513 seed24734 0.700
hellaswag/8522 seed24734 0.700
hellaswag/3196 seed24734 0.700
hellaswag/5221 seed24734 0.700
hellaswag/7356 seed24734 0.700
hellaswag/7177 seed24734 0.700
hellaswag/522 seed24734 0.700
hellaswag/6964 seed24734 0.700
hellaswag/658 seed24734 0.700
hellaswag/9527 seed24734 0.700
hellaswag/9759 seed24734 0.700
hellaswag/6341 seed24734 0.700
hellaswag/9088 seed24734 0.700
hellaswag/2004 seed24734 0.700
hellaswag/893 seed24734 0.700
hellaswag/1283 seed24734 0.700
hellaswag/1455 seed24734 0.700
hellaswag/1252 seed6115 0.700
hellaswag/1328 seed6115 0.700
hellaswag/1450 seed6115 0.700
hellaswag/1505 seed6115 0.700
hellaswag/6092 seed24762 0.700
hellaswag/879 seed24762 0.700
hellaswag/9536 seed24762 0.700
hellaswag/9700 seed24762 0.700
hellaswag/9353 seed6115 0.700
hellaswag/9147 seed6115 0.700
hellaswag/8850 seed6115 0.700
hellaswag/8849 seed24762 0.700
hellaswag/5978 seed6115 0.700
hellaswag/4879 seed6115 0.700
hellaswag/6596 seed24762 0.700
hellaswag/6399 seed24762 0.700
hellaswag/9746 seed6115 0.700
hellaswag/6029 seed24762 0.700
hellaswag/8043 seed6115 0.700
hellaswag/7954 seed24762 0.700
hellaswag/8640 seed24762 0.700
hellaswag/3494 seed6115 0.700
hellaswag/7383 seed6115 0.700
hellaswag/4777 seed6115 0.700
hellaswag/7837 seed6115 0.700
hellaswag/2865 seed6115 0.700
hellaswag/7805 seed24762 0.700
hellaswag/7092 seed24762 0.700
hellaswag/4806 seed6115 0.700
hellaswag/5790 seed6115 0.700
hellaswag/4027 seed6115 0.700
hellaswag/3949 seed24762 0.700
hellaswag/5519 seed6115 0.700
hellaswag/5375 seed6115 0.700
hellaswag/5226 seed24762 0.700
hellaswag/5225 seed24762 0.700
hellaswag/993 seed6115 0.700
hellaswag/2511 seed24762 0.700
hellaswag/8324 seed6115 0.700
hellaswag/8312 seed6115 0.700
hellaswag/8409 seed24762 0.700
hellaswag/852 seed6115 0.700
hellaswag/8485 seed24762 0.700
hellaswag/8416 seed6115 0.700
hellaswag/2463 seed6115 0.700
hellaswag/2460 seed24762 0.700
hellaswag/2590 seed24762 0.700
hellaswag/264 seed6115 0.700
hellaswag/3352 seed24762 0.700
hellaswag/322 seed17495 0.700
hellaswag/5605 seed17495 0.700
hellaswag/5057 seed17495 0.700
hellaswag/1697 seed17495 0.700
hellaswag/1486 seed17495 0.700
hellaswag/8245 seed17495 0.700
hellaswag/2741 seed17495 0.700
hellaswag/3317 seed17495 0.700
hellaswag/3248 seed17495 0.700
hellaswag/2688 seed17495 0.700
hellaswag/2515 seed17495 0.700
hellaswag/1024 seed17495 0.700
hellaswag/629 seed17495 0.700
hellaswag/9172 seed17495 0.700
hellaswag/1841 seed17495 0.700
hellaswag/4930 seed17495 0.700
hellaswag/6874 seed17495 0.700
hellaswag/6306 seed17495 0.700
hellaswag/4000 seed17495 0.700
hellaswag/4731 seed17495 0.700
hellaswag/8421 seed17495 0.700
hellaswag/7420 seed17495 0.700
hellaswag/2855 seed17495 0.700
hellaswag/6650 seed30841 0.699
hellaswag/1968 seed30841 0.699
hellaswag/1944 seed30841 0.699
hellaswag/9334 seed30841 0.699
hellaswag/1870 seed30841 0.699
hellaswag/659 seed30841 0.699
hellaswag/6844 seed30841 0.699
hellaswag/6415 seed30841 0.699
hellaswag/1990 seed30841 0.699
hellaswag/4156 seed30841 0.699
hellaswag/5280 seed30841 0.699
hellaswag/5675 seed30841 0.699
hellaswag/3130 seed30841 0.699
hellaswag/3030 seed30841 0.699
hellaswag/3011 seed30841 0.699
hellaswag/7102 seed30841 0.699
hellaswag/6890 seed30841 0.699
hellaswag/4773 seed30841 0.699
hellaswag/427 seed30841 0.699
hellaswag/4401 seed30841 0.699
hellaswag/410 seed30841 0.699
hellaswag/5739 seed30841 0.699
hellaswag/3406 seed30841 0.699
hellaswag/3505 seed30841 0.699
hellaswag/2736 seed30841 0.699
hellaswag/8284 seed30841 0.699
hellaswag/8348 seed30841 0.699
hellaswag/1046 seed30841 0.699

Suspect problems

These are 10 problems with the lowest correlation with the overall evaluation (i.e. better models tend to do worse on these. )

example_link pass1_of_ex tau
hellaswag/8001 0.500 -0.754
hellaswag/5906 0.600 -0.739
hellaswag/4586 0.400 -0.739
hellaswag/2052 0.700 -0.691
hellaswag/7541 0.600 -0.677
hellaswag/8664 0.300 -0.658
hellaswag/2823 0.300 -0.658
hellaswag/780 0.300 -0.658
hellaswag/4342 0.300 -0.658
hellaswag/4469 0.300 -0.658

Histogram of accuracies

Histogram of problems by the accuracy on each problem.

Histogram of difficulties

Histogram of problems by the minimum win rate to solve each problem.