There are 1047 examples not solved by any model.
Solving some of these can be a good signal that your model is indeed better than leading models if these are good problems.
gsm8k/10, gsm8k/100, gsm8k/1000, gsm8k/1001, gsm8k/1002, gsm8k/1003, gsm8k/1004, gsm8k/1005, gsm8k/1006, gsm8k/1007, gsm8k/1008, gsm8k/1009, gsm8k/101, gsm8k/1010, gsm8k/1011, gsm8k/1012, gsm8k/1013, gsm8k/1014, gsm8k/1015, gsm8k/1016, gsm8k/1017, gsm8k/1018, gsm8k/1019, gsm8k/102, gsm8k/1020, gsm8k/1021, gsm8k/1022, gsm8k/1023, gsm8k/1024, gsm8k/1026, gsm8k/1027, gsm8k/1029, gsm8k/103, gsm8k/1030, gsm8k/1031, gsm8k/1033, gsm8k/1035, gsm8k/1036, gsm8k/1037, gsm8k/1039, gsm8k/104, gsm8k/1040, gsm8k/1041, gsm8k/1042, gsm8k/1043, gsm8k/1045, gsm8k/1046, gsm8k/1048, gsm8k/1049, gsm8k/1050, gsm8k/1052, gsm8k/1055, gsm8k/1056, gsm8k/1057, gsm8k/1058, gsm8k/106, gsm8k/1060, gsm8k/1061, gsm8k/1062, gsm8k/1063, gsm8k/1064, gsm8k/1065, gsm8k/1066, gsm8k/1067, gsm8k/1069, gsm8k/107, gsm8k/1070, gsm8k/1071, gsm8k/1072, gsm8k/1073, gsm8k/1075, gsm8k/1076, gsm8k/1077, gsm8k/1078, gsm8k/1079, gsm8k/108, gsm8k/1080, gsm8k/1081, gsm8k/1083, gsm8k/1084, gsm8k/1085, gsm8k/1086, gsm8k/1087, gsm8k/1088, gsm8k/1089, gsm8k/109, gsm8k/1090, gsm8k/1092, gsm8k/1093, gsm8k/1094, gsm8k/1095, gsm8k/1096, gsm8k/1097, gsm8k/1099, gsm8k/11, gsm8k/1100, gsm8k/1101, gsm8k/1102, gsm8k/1103, gsm8k/1105, gsm8k/1107, gsm8k/1110, gsm8k/1111, gsm8k/1112, gsm8k/1113, gsm8k/1114, gsm8k/1117, gsm8k/1118, gsm8k/1119, gsm8k/1121, gsm8k/1122, gsm8k/1123, gsm8k/1124, gsm8k/1125, gsm8k/1126, gsm8k/1128, gsm8k/1129, gsm8k/1130, gsm8k/1131, gsm8k/1132, gsm8k/1133, gsm8k/1134, gsm8k/1135, gsm8k/1136, gsm8k/1137, gsm8k/1138, gsm8k/1139, gsm8k/114, gsm8k/1140, gsm8k/1141, gsm8k/1142, gsm8k/1144, gsm8k/1145, gsm8k/1146, gsm8k/1147, gsm8k/1148, gsm8k/1149, gsm8k/115, gsm8k/1150, gsm8k/1151, gsm8k/1152, gsm8k/1153, gsm8k/1154, gsm8k/1155, gsm8k/1156, gsm8k/1158, gsm8k/1159, gsm8k/116, gsm8k/1160, gsm8k/1161, gsm8k/1162, gsm8k/1165, gsm8k/1166, gsm8k/1167, gsm8k/1168, gsm8k/1171, gsm8k/1172, gsm8k/1173, gsm8k/1174, gsm8k/1177, gsm8k/1178, gsm8k/1179, gsm8k/118, gsm8k/1180, gsm8k/1181, gsm8k/1182, gsm8k/1184, gsm8k/1185, gsm8k/1186, gsm8k/1187, gsm8k/1188, gsm8k/1189, gsm8k/119, gsm8k/1190, gsm8k/1192, gsm8k/1193, gsm8k/1194, gsm8k/1195, gsm8k/1198, gsm8k/1199, gsm8k/12, gsm8k/120, gsm8k/1200, gsm8k/1201, gsm8k/1202, gsm8k/1204, gsm8k/1205, gsm8k/1207, gsm8k/1208, gsm8k/1209, gsm8k/121, gsm8k/1211, gsm8k/1213, gsm8k/1216, gsm8k/1217, gsm8k/1218, gsm8k/122, gsm8k/1220, gsm8k/1221, gsm8k/1223, gsm8k/1226, gsm8k/1227, gsm8k/1228, gsm8k/1229, gsm8k/123, gsm8k/1230, gsm8k/1233, gsm8k/1235, gsm8k/1236, gsm8k/1237, gsm8k/1238, gsm8k/1239, gsm8k/124, gsm8k/1240, gsm8k/1241, gsm8k/1242, gsm8k/1243, gsm8k/1244, gsm8k/1246, gsm8k/1248, gsm8k/1249, gsm8k/1250, gsm8k/1251, gsm8k/1252, gsm8k/1253, gsm8k/1255, gsm8k/1256, gsm8k/1257, gsm8k/1259, gsm8k/126, gsm8k/1260, gsm8k/1261, gsm8k/1262, gsm8k/1263, gsm8k/1264, gsm8k/1265, gsm8k/1266, gsm8k/1267, gsm8k/1268, gsm8k/1269, gsm8k/127, gsm8k/1270, gsm8k/1271, gsm8k/1272, gsm8k/1273, gsm8k/1274, gsm8k/1275, gsm8k/1276, gsm8k/1278, gsm8k/1279, gsm8k/128, gsm8k/1280, gsm8k/1283, gsm8k/1284, gsm8k/1285, gsm8k/1286, gsm8k/1287, gsm8k/1288, gsm8k/1289, gsm8k/129, gsm8k/1290, gsm8k/1292, gsm8k/1293, gsm8k/1294, gsm8k/1295, gsm8k/1296, gsm8k/1298, gsm8k/1299, gsm8k/13, gsm8k/130, gsm8k/1300, gsm8k/1301, gsm8k/1303, gsm8k/1305, gsm8k/1307, gsm8k/1309, gsm8k/1310, gsm8k/1312, gsm8k/1313, gsm8k/1314, gsm8k/1315, gsm8k/1316, gsm8k/1317, gsm8k/1318, gsm8k/132, gsm8k/133, gsm8k/135, gsm8k/136, gsm8k/137, gsm8k/138, gsm8k/139, gsm8k/14, gsm8k/141, gsm8k/144, gsm8k/145, gsm8k/146, gsm8k/147, gsm8k/148, gsm8k/15, gsm8k/150, gsm8k/151, gsm8k/153, gsm8k/154, gsm8k/155, gsm8k/156, gsm8k/157, gsm8k/158, gsm8k/159, gsm8k/160, gsm8k/161, gsm8k/162, gsm8k/163, gsm8k/166, gsm8k/167, gsm8k/168, gsm8k/169, gsm8k/17, gsm8k/171, gsm8k/172, gsm8k/173, gsm8k/174, gsm8k/175, gsm8k/177, gsm8k/178, gsm8k/18, gsm8k/181, gsm8k/182, gsm8k/183, gsm8k/184, gsm8k/186, gsm8k/187, gsm8k/189, gsm8k/191, gsm8k/192, gsm8k/193, gsm8k/195, gsm8k/196, gsm8k/197, gsm8k/198, gsm8k/199, gsm8k/20, gsm8k/201, gsm8k/203, gsm8k/204, gsm8k/205, gsm8k/206, gsm8k/207, gsm8k/208, gsm8k/209, gsm8k/21, gsm8k/211, gsm8k/212, gsm8k/213, gsm8k/215, gsm8k/216, gsm8k/218, gsm8k/219, gsm8k/220, gsm8k/221, gsm8k/222, gsm8k/226, gsm8k/227, gsm8k/228, gsm8k/229, gsm8k/23, gsm8k/230, gsm8k/234, gsm8k/236, gsm8k/238, gsm8k/239, gsm8k/24, gsm8k/240, gsm8k/241, gsm8k/242, gsm8k/243, gsm8k/244, gsm8k/245, gsm8k/246, gsm8k/247, gsm8k/249, gsm8k/25, gsm8k/250, gsm8k/252, gsm8k/254, gsm8k/255, gsm8k/256, gsm8k/257, gsm8k/259, gsm8k/26, gsm8k/260, gsm8k/261, gsm8k/262, gsm8k/263, gsm8k/264, gsm8k/265, gsm8k/266, gsm8k/267, gsm8k/27, gsm8k/270, gsm8k/271, gsm8k/272, gsm8k/273, gsm8k/275, gsm8k/276, gsm8k/277, gsm8k/279, gsm8k/280, gsm8k/281, gsm8k/282, gsm8k/283, gsm8k/286, gsm8k/287, gsm8k/29, gsm8k/290, gsm8k/291, gsm8k/292, gsm8k/293, gsm8k/295, gsm8k/296, gsm8k/297, gsm8k/298, gsm8k/299, gsm8k/30, gsm8k/301, gsm8k/303, gsm8k/304, gsm8k/306, gsm8k/307, gsm8k/308, gsm8k/309, gsm8k/31, gsm8k/310, gsm8k/311, gsm8k/312, gsm8k/313, gsm8k/314, gsm8k/315, gsm8k/316, gsm8k/318, gsm8k/320, gsm8k/322, gsm8k/323, gsm8k/324, gsm8k/325, gsm8k/326, gsm8k/328, gsm8k/33, gsm8k/330, gsm8k/331, gsm8k/332, gsm8k/333, gsm8k/334, gsm8k/336, gsm8k/337, gsm8k/338, gsm8k/339, gsm8k/34, gsm8k/340, gsm8k/341, gsm8k/342, gsm8k/343, gsm8k/345, gsm8k/346, gsm8k/347, gsm8k/349, gsm8k/35, gsm8k/350, gsm8k/353, gsm8k/354, gsm8k/355, gsm8k/356, gsm8k/357, gsm8k/358, gsm8k/359, gsm8k/36, gsm8k/361, gsm8k/362, gsm8k/363, gsm8k/364, gsm8k/367, gsm8k/368, gsm8k/369, gsm8k/37, gsm8k/371, gsm8k/374, gsm8k/377, gsm8k/378, gsm8k/38, gsm8k/380, gsm8k/383, gsm8k/384, gsm8k/385, gsm8k/388, gsm8k/389, gsm8k/39, gsm8k/390, gsm8k/391, gsm8k/392, gsm8k/393, gsm8k/394, gsm8k/395, gsm8k/397, gsm8k/398, gsm8k/399, gsm8k/401, gsm8k/403, gsm8k/404, gsm8k/405, gsm8k/406, gsm8k/408, gsm8k/409, gsm8k/41, gsm8k/410, gsm8k/411, gsm8k/412, gsm8k/413, gsm8k/414, gsm8k/415, gsm8k/416, gsm8k/417, gsm8k/418, gsm8k/420, gsm8k/421, gsm8k/422, gsm8k/423, gsm8k/424, gsm8k/425, gsm8k/426, gsm8k/428, gsm8k/429, gsm8k/43, gsm8k/430, gsm8k/431, gsm8k/432, gsm8k/433, gsm8k/436, gsm8k/437, gsm8k/439, gsm8k/440, gsm8k/442, gsm8k/444, gsm8k/445, gsm8k/446, gsm8k/447, gsm8k/448, gsm8k/449, gsm8k/45, gsm8k/450, gsm8k/451, gsm8k/452, gsm8k/454, gsm8k/456, gsm8k/457, gsm8k/458, gsm8k/46, gsm8k/460, gsm8k/461, gsm8k/462, gsm8k/463, gsm8k/464, gsm8k/465, gsm8k/466, gsm8k/471, gsm8k/473, gsm8k/474, gsm8k/475, gsm8k/476, gsm8k/477, gsm8k/478, gsm8k/479, gsm8k/48, gsm8k/480, gsm8k/481, gsm8k/484, gsm8k/485, gsm8k/486, gsm8k/487, gsm8k/488, gsm8k/489, gsm8k/490, gsm8k/493, gsm8k/494, gsm8k/496, gsm8k/498, gsm8k/499, gsm8k/5, gsm8k/50, gsm8k/500, gsm8k/501, gsm8k/502, gsm8k/503, gsm8k/504, gsm8k/505, gsm8k/507, gsm8k/508, gsm8k/509, gsm8k/51, gsm8k/510, gsm8k/513, gsm8k/514, gsm8k/515, gsm8k/518, gsm8k/519, gsm8k/52, gsm8k/520, gsm8k/521, gsm8k/522, gsm8k/523, gsm8k/524, gsm8k/525, gsm8k/526, gsm8k/527, gsm8k/528, gsm8k/529, gsm8k/530, gsm8k/531, gsm8k/533, gsm8k/534, gsm8k/535, gsm8k/537, gsm8k/538, gsm8k/54, gsm8k/541, gsm8k/542, gsm8k/544, gsm8k/545, gsm8k/546, gsm8k/547, gsm8k/548, gsm8k/549, gsm8k/550, gsm8k/552, gsm8k/553, gsm8k/555, gsm8k/557, gsm8k/559, gsm8k/56, gsm8k/560, gsm8k/561, gsm8k/562, gsm8k/563, gsm8k/564, gsm8k/565, gsm8k/566, gsm8k/568, gsm8k/57, gsm8k/570, gsm8k/571, gsm8k/572, gsm8k/573, gsm8k/574, gsm8k/575, gsm8k/576, gsm8k/577, gsm8k/58, gsm8k/580, gsm8k/581, gsm8k/582, gsm8k/583, gsm8k/586, gsm8k/587, gsm8k/589, gsm8k/59, gsm8k/590, gsm8k/591, gsm8k/592, gsm8k/593, gsm8k/594, gsm8k/596, gsm8k/598, gsm8k/599, gsm8k/6, gsm8k/60, gsm8k/600, gsm8k/601, gsm8k/602, gsm8k/603, gsm8k/604, gsm8k/605, gsm8k/606, gsm8k/607, gsm8k/608, gsm8k/609, gsm8k/61, gsm8k/610, gsm8k/611, gsm8k/612, gsm8k/613, gsm8k/614, gsm8k/615, gsm8k/616, gsm8k/617, gsm8k/618, gsm8k/619, gsm8k/62, gsm8k/620, gsm8k/621, gsm8k/622, gsm8k/625, gsm8k/626, gsm8k/627, gsm8k/629, gsm8k/63, gsm8k/631, gsm8k/632, gsm8k/634, gsm8k/635, gsm8k/636, gsm8k/638, gsm8k/639, gsm8k/64, gsm8k/640, gsm8k/641, gsm8k/642, gsm8k/644, gsm8k/647, gsm8k/648, gsm8k/649, gsm8k/65, gsm8k/650, gsm8k/651, gsm8k/652, gsm8k/653, gsm8k/655, gsm8k/656, gsm8k/657, gsm8k/658, gsm8k/659, gsm8k/66, gsm8k/660, gsm8k/661, gsm8k/662, gsm8k/664, gsm8k/666, gsm8k/667, gsm8k/669, gsm8k/67, gsm8k/671, gsm8k/672, gsm8k/673, gsm8k/674, gsm8k/675, gsm8k/676, gsm8k/677, gsm8k/679, gsm8k/680, gsm8k/681, gsm8k/682, gsm8k/687, gsm8k/688, gsm8k/689, gsm8k/690, gsm8k/692, gsm8k/693, gsm8k/694, gsm8k/696, gsm8k/698, gsm8k/699, gsm8k/70, gsm8k/700, gsm8k/701, gsm8k/702, gsm8k/703, gsm8k/706, gsm8k/707, gsm8k/709, gsm8k/711, gsm8k/712, gsm8k/713, gsm8k/714, gsm8k/715, gsm8k/716, gsm8k/717, gsm8k/718, gsm8k/72, gsm8k/720, gsm8k/721, gsm8k/722, gsm8k/723, gsm8k/724, gsm8k/725, gsm8k/726, gsm8k/727, gsm8k/729, gsm8k/73, gsm8k/730, gsm8k/731, gsm8k/732, gsm8k/734, gsm8k/735, gsm8k/736, gsm8k/737, gsm8k/739, gsm8k/74, gsm8k/740, gsm8k/742, gsm8k/745, gsm8k/747, gsm8k/748, gsm8k/749, gsm8k/75, gsm8k/750, gsm8k/751, gsm8k/752, gsm8k/753, gsm8k/754, gsm8k/755, gsm8k/756, gsm8k/757, gsm8k/759, gsm8k/76, gsm8k/760, gsm8k/761, gsm8k/762, gsm8k/763, gsm8k/764, gsm8k/765, gsm8k/766, gsm8k/768, gsm8k/769, gsm8k/77, gsm8k/770, gsm8k/771, gsm8k/772, gsm8k/773, gsm8k/774, gsm8k/775, gsm8k/777, gsm8k/778, gsm8k/779, gsm8k/78, gsm8k/780, gsm8k/781, gsm8k/782, gsm8k/785, gsm8k/787, gsm8k/789, gsm8k/790, gsm8k/793, gsm8k/794, gsm8k/795, gsm8k/796, gsm8k/797, gsm8k/8, gsm8k/800, gsm8k/801, gsm8k/802, gsm8k/803, gsm8k/805, gsm8k/806, gsm8k/807, gsm8k/808, gsm8k/81, gsm8k/810, gsm8k/811, gsm8k/812, gsm8k/813, gsm8k/814, gsm8k/82, gsm8k/820, gsm8k/822, gsm8k/823, gsm8k/824, gsm8k/825, gsm8k/826, gsm8k/827, gsm8k/828, gsm8k/829, gsm8k/830, gsm8k/831, gsm8k/832, gsm8k/833, gsm8k/834, gsm8k/835, gsm8k/836, gsm8k/837, gsm8k/838, gsm8k/84, gsm8k/840, gsm8k/841, gsm8k/842, gsm8k/843, gsm8k/844, gsm8k/845, gsm8k/846, gsm8k/847, gsm8k/848, gsm8k/849, gsm8k/85, gsm8k/850, gsm8k/851, gsm8k/852, gsm8k/853, gsm8k/854, gsm8k/855, gsm8k/856, gsm8k/857, gsm8k/858, gsm8k/859, gsm8k/86, gsm8k/861, gsm8k/862, gsm8k/863, gsm8k/864, gsm8k/865, gsm8k/866, gsm8k/867, gsm8k/868, gsm8k/869, gsm8k/87, gsm8k/870, gsm8k/871, gsm8k/872, gsm8k/873, gsm8k/874, gsm8k/875, gsm8k/876, gsm8k/877, gsm8k/878, gsm8k/879, gsm8k/88, gsm8k/880, gsm8k/881, gsm8k/882, gsm8k/884, gsm8k/885, gsm8k/886, gsm8k/887, gsm8k/888, gsm8k/889, gsm8k/890, gsm8k/891, gsm8k/892, gsm8k/893, gsm8k/894, gsm8k/895, gsm8k/896, gsm8k/898, gsm8k/899, gsm8k/9, gsm8k/900, gsm8k/901, gsm8k/902, gsm8k/903, gsm8k/904, gsm8k/905, gsm8k/906, gsm8k/907, gsm8k/908, gsm8k/909, gsm8k/91, gsm8k/910, gsm8k/911, gsm8k/912, gsm8k/913, gsm8k/914, gsm8k/915, gsm8k/916, gsm8k/917, gsm8k/918, gsm8k/919, gsm8k/92, gsm8k/920, gsm8k/921, gsm8k/922, gsm8k/923, gsm8k/924, gsm8k/925, gsm8k/926, gsm8k/927, gsm8k/928, gsm8k/929, gsm8k/930, gsm8k/931, gsm8k/932, gsm8k/933, gsm8k/934, gsm8k/935, gsm8k/936, gsm8k/937, gsm8k/938, gsm8k/939, gsm8k/94, gsm8k/940, gsm8k/941, gsm8k/942, gsm8k/943, gsm8k/944, gsm8k/945, gsm8k/946, gsm8k/947, gsm8k/948, gsm8k/949, gsm8k/95, gsm8k/950, gsm8k/951, gsm8k/952, gsm8k/953, gsm8k/954, gsm8k/955, gsm8k/956, gsm8k/957, gsm8k/958, gsm8k/959, gsm8k/960, gsm8k/961, gsm8k/962, gsm8k/963, gsm8k/964, gsm8k/965, gsm8k/966, gsm8k/967, gsm8k/968, gsm8k/969, gsm8k/97, gsm8k/970, gsm8k/971, gsm8k/972, gsm8k/973, gsm8k/974, gsm8k/975, gsm8k/976, gsm8k/977, gsm8k/978, gsm8k/979, gsm8k/98, gsm8k/980, gsm8k/981, gsm8k/982, gsm8k/983, gsm8k/984, gsm8k/985, gsm8k/986, gsm8k/987, gsm8k/988, gsm8k/989, gsm8k/99, gsm8k/990, gsm8k/991, gsm8k/992, gsm8k/993, gsm8k/994, gsm8k/995, gsm8k/996, gsm8k/997, gsm8k/998, gsm8k/999
| example_link | model | min_pass1_of_model |
|---|---|---|
| gsm8k/1038 | seed27712 | 0.050 |
| gsm8k/1127 | seed27712 | 0.050 |
| gsm8k/1053 | seed27712 | 0.050 |
| gsm8k/90 | seed27712 | 0.050 |
| gsm8k/746 | seed27712 | 0.050 |
| gsm8k/42 | seed27712 | 0.050 |
| gsm8k/453 | seed27712 | 0.050 |
| gsm8k/472 | seed27712 | 0.050 |
| gsm8k/497 | seed27712 | 0.050 |
| gsm8k/643 | seed27712 | 0.050 |
| gsm8k/623 | seed27712 | 0.050 |
| gsm8k/744 | seed27712 | 0.050 |
| gsm8k/285 | seed27712 | 0.050 |
| gsm8k/2 | seed27712 | 0.050 |
| gsm8k/268 | seed27712 | 0.050 |
| gsm8k/294 | seed27712 | 0.050 |
| gsm8k/164 | seed27712 | 0.050 |
| gsm8k/1281 | seed27712 | 0.050 |
| gsm8k/1302 | seed27712 | 0.050 |
| gsm8k/152 | seed27712 | 0.050 |
| gsm8k/1183 | seed27712 | 0.050 |
| gsm8k/695 | seed27712 | 0.050 |
| gsm8k/784 | seed27712 | 0.050 |
| gsm8k/482 | seed30841 | 0.047 |
| gsm8k/459 | seed30841 | 0.047 |
| gsm8k/578 | seed30841 | 0.047 |
| gsm8k/678 | seed30841 | 0.047 |
| gsm8k/705 | seed17495 | 0.047 |
| gsm8k/663 | seed30841 | 0.047 |
| gsm8k/788 | seed17495 | 0.047 |
| gsm8k/1143 | seed30841 | 0.047 |
| gsm8k/1074 | seed30841 | 0.047 |
| gsm8k/1091 | seed30841 | 0.047 |
| gsm8k/1059 | seed17495 | 0.047 |
| gsm8k/1068 | seed30841 | 0.047 |
| gsm8k/1108 | seed17495 | 0.047 |
| gsm8k/1164 | seed17495 | 0.047 |
| gsm8k/1032 | seed17495 | 0.047 |
| gsm8k/1044 | seed30841 | 0.047 |
| gsm8k/468 | seed17495 | 0.047 |
| gsm8k/539 | seed17495 | 0.047 |
| gsm8k/366 | seed17495 | 0.047 |
| gsm8k/317 | seed30841 | 0.047 |
| gsm8k/344 | seed30841 | 0.047 |
| gsm8k/335 | seed30841 | 0.047 |
| gsm8k/300 | seed17495 | 0.047 |
| gsm8k/185 | seed30841 | 0.047 |
| gsm8k/284 | seed30841 | 0.047 |
| gsm8k/28 | seed30841 | 0.047 |
| gsm8k/125 | seed30841 | 0.047 |
| gsm8k/1170 | seed17495 | 0.047 |
| gsm8k/1176 | seed30841 | 0.047 |
| gsm8k/1212 | seed17495 | 0.047 |
| gsm8k/407 | seed30841 | 0.047 |
| gsm8k/467 | seed30841 | 0.047 |
| gsm8k/470 | seed30841 | 0.047 |
| gsm8k/49 | seed17495 | 0.047 |
| gsm8k/567 | seed17495 | 0.047 |
| gsm8k/351 | seed30841 | 0.047 |
| gsm8k/387 | seed17495 | 0.047 |
| gsm8k/348 | seed30841 | 0.047 |
| gsm8k/258 | seed17495 | 0.047 |
| gsm8k/816 | seed17495 | 0.047 |
| gsm8k/860 | seed17495 | 0.047 |
| gsm8k/839 | seed30841 | 0.047 |
| gsm8k/80 | seed17495 | 0.047 |
| gsm8k/798 | seed30841 | 0.047 |
| gsm8k/710 | seed30841 | 0.047 |
| gsm8k/646 | seed17495 | 0.047 |
| gsm8k/540 | seed24762 | 0.042 |
| gsm8k/47 | seed24762 | 0.042 |
| gsm8k/419 | seed24762 | 0.042 |
| gsm8k/670 | seed24762 | 0.042 |
| gsm8k/1297 | seed24762 | 0.042 |
| gsm8k/1191 | seed24762 | 0.042 |
| gsm8k/143 | seed24762 | 0.042 |
| gsm8k/165 | seed24762 | 0.042 |
| gsm8k/1120 | seed24762 | 0.042 |
| gsm8k/396 | seed24762 | 0.042 |
| gsm8k/140 | seed24762 | 0.042 |
| gsm8k/1254 | seed24762 | 0.042 |
| gsm8k/883 | seed6529 | 0.041 |
| gsm8k/815 | seed6529 | 0.041 |
| gsm8k/1047 | seed6529 | 0.041 |
| gsm8k/1104 | seed6529 | 0.041 |
| gsm8k/741 | seed6529 | 0.041 |
| gsm8k/495 | seed6529 | 0.041 |
| gsm8k/551 | seed6529 | 0.041 |
| gsm8k/382 | seed6529 | 0.041 |
| gsm8k/190 | seed6529 | 0.041 |
| gsm8k/149 | seed6529 | 0.041 |
| gsm8k/791 | seed6529 | 0.041 |
| gsm8k/68 | seed6529 | 0.041 |
| gsm8k/170 | seed6529 | 0.041 |
| gsm8k/786 | seed14309 | 0.039 |
| gsm8k/665 | seed14309 | 0.039 |
| gsm8k/897 | seed6115 | 0.039 |
| gsm8k/1169 | seed6115 | 0.039 |
| gsm8k/111 | seed14309 | 0.039 |
| gsm8k/818 | seed6115 | 0.039 |
| gsm8k/532 | seed14309 | 0.039 |
| gsm8k/783 | seed14309 | 0.039 |
| gsm8k/758 | seed14309 | 0.039 |
| gsm8k/253 | seed14309 | 0.039 |
| gsm8k/1175 | seed6115 | 0.039 |
| gsm8k/1197 | seed6115 | 0.039 |
| gsm8k/1304 | seed6115 | 0.039 |
| gsm8k/1224 | seed6115 | 0.039 |
| gsm8k/1231 | seed14309 | 0.039 |
| gsm8k/4 | seed14309 | 0.039 |
| gsm8k/40 | seed6115 | 0.039 |
| gsm8k/327 | seed14309 | 0.039 |
| gsm8k/269 | seed6115 | 0.039 |
| gsm8k/427 | seed14309 | 0.039 |
| gsm8k/704 | seed14309 | 0.039 |
| gsm8k/558 | seed6115 | 0.039 |
| gsm8k/637 | seed14309 | 0.039 |
| gsm8k/686 | seed16385 | 0.038 |
| gsm8k/585 | seed16385 | 0.038 |
| gsm8k/506 | seed16385 | 0.038 |
| gsm8k/288 | seed16385 | 0.038 |
| gsm8k/142 | seed16385 | 0.038 |
| gsm8k/1203 | seed16385 | 0.038 |
| gsm8k/443 | seed16385 | 0.038 |
| gsm8k/441 | seed16385 | 0.038 |
| gsm8k/274 | seed16385 | 0.038 |
| gsm8k/402 | seed16385 | 0.038 |
| gsm8k/376 | seed16385 | 0.038 |
| gsm8k/375 | seed16385 | 0.038 |
| gsm8k/1306 | seed149 | 0.035 |
| gsm8k/117 | seed149 | 0.035 |
| gsm8k/110 | seed149 | 0.035 |
| gsm8k/0 | seed149 | 0.035 |
| gsm8k/728 | seed149 | 0.035 |
| gsm8k/1225 | seed149 | 0.035 |
| gsm8k/1196 | seed149 | 0.035 |
| gsm8k/685 | seed149 | 0.035 |
| gsm8k/305 | seed149 | 0.035 |
| gsm8k/233 | seed149 | 0.035 |
| gsm8k/235 | seed149 | 0.035 |
| gsm8k/1115 | seed24734 | 0.033 |
| gsm8k/372 | seed24734 | 0.033 |
| gsm8k/289 | seed24734 | 0.033 |
| gsm8k/1215 | seed24734 | 0.033 |
| gsm8k/1219 | seed24734 | 0.033 |
| gsm8k/1245 | seed24734 | 0.033 |
| gsm8k/455 | seed24734 | 0.033 |
| gsm8k/554 | seed24734 | 0.033 |
| gsm8k/691 | seed24734 | 0.033 |
These are 10 problems with the lowest correlation with the overall evaluation (i.e. better models tend to do worse on these. )
| example_link | pass1_of_ex | tau |
|---|---|---|
| gsm8k/251 | 0.400 | -0.623 |
| gsm8k/1098 | 0.300 | -0.599 |
| gsm8k/22 | 0.200 | -0.534 |
| gsm8k/630 | 0.200 | -0.534 |
| gsm8k/691 | 0.100 | -0.457 |
| gsm8k/1219 | 0.100 | -0.457 |
| gsm8k/1245 | 0.100 | -0.457 |
| gsm8k/1034 | 0.200 | -0.457 |
| gsm8k/329 | 0.900 | -0.457 |
| gsm8k/289 | 0.100 | -0.457 |
Histogram of problems by the accuracy on each problem.
Histogram of problems by the minimum win rate to solve each problem.