There are 1226 examples not solved by any model.
Solving some of these can be a good signal that your model is indeed better than leading models if these are good problems.
gsm8k/0, gsm8k/1, gsm8k/10, gsm8k/100, gsm8k/1000, gsm8k/1001, gsm8k/1002, gsm8k/1003, gsm8k/1004, gsm8k/1005, gsm8k/1006, gsm8k/1007, gsm8k/1008, gsm8k/1009, gsm8k/101, gsm8k/1010, gsm8k/1011, gsm8k/1012, gsm8k/1013, gsm8k/1014, gsm8k/1015, gsm8k/1016, gsm8k/1017, gsm8k/1018, gsm8k/1019, gsm8k/102, gsm8k/1020, gsm8k/1021, gsm8k/1022, gsm8k/1023, gsm8k/1024, gsm8k/1026, gsm8k/1027, gsm8k/1028, gsm8k/1029, gsm8k/103, gsm8k/1030, gsm8k/1031, gsm8k/1032, gsm8k/1033, gsm8k/1034, gsm8k/1035, gsm8k/1039, gsm8k/104, gsm8k/1040, gsm8k/1041, gsm8k/1042, gsm8k/1043, gsm8k/1044, gsm8k/1045, gsm8k/1046, gsm8k/1048, gsm8k/1049, gsm8k/105, gsm8k/1050, gsm8k/1051, gsm8k/1052, gsm8k/1053, gsm8k/1054, gsm8k/1055, gsm8k/1056, gsm8k/1057, gsm8k/1058, gsm8k/1059, gsm8k/106, gsm8k/1060, gsm8k/1061, gsm8k/1062, gsm8k/1063, gsm8k/1064, gsm8k/1065, gsm8k/1066, gsm8k/1067, gsm8k/1068, gsm8k/1069, gsm8k/107, gsm8k/1070, gsm8k/1071, gsm8k/1072, gsm8k/1073, gsm8k/1074, gsm8k/1075, gsm8k/1076, gsm8k/1077, gsm8k/1078, gsm8k/108, gsm8k/1080, gsm8k/1081, gsm8k/1082, gsm8k/1083, gsm8k/1084, gsm8k/1085, gsm8k/1086, gsm8k/1087, gsm8k/1088, gsm8k/1089, gsm8k/109, gsm8k/1090, gsm8k/1091, gsm8k/1092, gsm8k/1093, gsm8k/1094, gsm8k/1095, gsm8k/1096, gsm8k/1097, gsm8k/1098, gsm8k/1099, gsm8k/11, gsm8k/110, gsm8k/1101, gsm8k/1102, gsm8k/1103, gsm8k/1104, gsm8k/1105, gsm8k/1107, gsm8k/1108, gsm8k/1109, gsm8k/111, gsm8k/1110, gsm8k/1111, gsm8k/1112, gsm8k/1113, gsm8k/1115, gsm8k/1116, gsm8k/1117, gsm8k/1119, gsm8k/112, gsm8k/1120, gsm8k/1121, gsm8k/1122, gsm8k/1123, gsm8k/1124, gsm8k/1125, gsm8k/1126, gsm8k/1127, gsm8k/1128, gsm8k/1129, gsm8k/113, gsm8k/1130, gsm8k/1131, gsm8k/1132, gsm8k/1133, gsm8k/1134, gsm8k/1135, gsm8k/1136, gsm8k/1137, gsm8k/1138, gsm8k/1139, gsm8k/114, gsm8k/1140, gsm8k/1141, gsm8k/1142, gsm8k/1143, gsm8k/1144, gsm8k/1145, gsm8k/1146, gsm8k/1147, gsm8k/1148, gsm8k/1149, gsm8k/115, gsm8k/1150, gsm8k/1151, gsm8k/1152, gsm8k/1153, gsm8k/1154, gsm8k/1155, gsm8k/1156, gsm8k/1157, gsm8k/1158, gsm8k/1159, gsm8k/116, gsm8k/1160, gsm8k/1161, gsm8k/1162, gsm8k/1163, gsm8k/1164, gsm8k/1165, gsm8k/1166, gsm8k/1167, gsm8k/1168, gsm8k/1169, gsm8k/117, gsm8k/1170, gsm8k/1171, gsm8k/1172, gsm8k/1174, gsm8k/1175, gsm8k/1176, gsm8k/1177, gsm8k/1178, gsm8k/1179, gsm8k/118, gsm8k/1180, gsm8k/1181, gsm8k/1182, gsm8k/1183, gsm8k/1184, gsm8k/1186, gsm8k/1187, gsm8k/1188, gsm8k/1189, gsm8k/119, gsm8k/1190, gsm8k/1191, gsm8k/1192, gsm8k/1193, gsm8k/1194, gsm8k/1195, gsm8k/1196, gsm8k/1197, gsm8k/1198, gsm8k/12, gsm8k/120, gsm8k/1200, gsm8k/1201, gsm8k/1202, gsm8k/1203, gsm8k/1204, gsm8k/1206, gsm8k/1207, gsm8k/1208, gsm8k/1209, gsm8k/121, gsm8k/1210, gsm8k/1211, gsm8k/1212, gsm8k/1213, gsm8k/1215, gsm8k/1217, gsm8k/1218, gsm8k/122, gsm8k/1220, gsm8k/1222, gsm8k/1223, gsm8k/1224, gsm8k/1225, gsm8k/1226, gsm8k/1227, gsm8k/1228, gsm8k/1229, gsm8k/123, gsm8k/1230, gsm8k/1231, gsm8k/1232, gsm8k/1233, gsm8k/1235, gsm8k/1236, gsm8k/1237, gsm8k/1238, gsm8k/1239, gsm8k/124, gsm8k/1240, gsm8k/1242, gsm8k/1243, gsm8k/1244, gsm8k/1245, gsm8k/1246, gsm8k/1247, gsm8k/1248, gsm8k/1249, gsm8k/125, gsm8k/1250, gsm8k/1251, gsm8k/1252, gsm8k/1253, gsm8k/1254, gsm8k/1255, gsm8k/1256, gsm8k/1257, gsm8k/1258, gsm8k/1259, gsm8k/126, gsm8k/1260, gsm8k/1261, gsm8k/1262, gsm8k/1263, gsm8k/1264, gsm8k/1265, gsm8k/1266, gsm8k/1267, gsm8k/1268, gsm8k/1269, gsm8k/127, gsm8k/1270, gsm8k/1271, gsm8k/1272, gsm8k/1273, gsm8k/1274, gsm8k/1275, gsm8k/1276, gsm8k/1277, gsm8k/1278, gsm8k/1279, gsm8k/128, gsm8k/1280, gsm8k/1281, gsm8k/1282, gsm8k/1283, gsm8k/1284, gsm8k/1285, gsm8k/1286, gsm8k/1287, gsm8k/1288, gsm8k/1289, gsm8k/129, gsm8k/1290, gsm8k/1291, gsm8k/1292, gsm8k/1293, gsm8k/1294, gsm8k/1295, gsm8k/1296, gsm8k/1297, gsm8k/1298, gsm8k/1299, gsm8k/13, gsm8k/130, gsm8k/1300, gsm8k/1301, gsm8k/1302, gsm8k/1303, gsm8k/1305, gsm8k/1306, gsm8k/1307, gsm8k/1308, gsm8k/1309, gsm8k/131, gsm8k/1310, gsm8k/1311, gsm8k/1312, gsm8k/1313, gsm8k/1314, gsm8k/1315, gsm8k/1316, gsm8k/1317, gsm8k/1318, gsm8k/132, gsm8k/133, gsm8k/134, gsm8k/135, gsm8k/136, gsm8k/137, gsm8k/138, gsm8k/139, gsm8k/14, gsm8k/140, gsm8k/141, gsm8k/142, gsm8k/143, gsm8k/144, gsm8k/145, gsm8k/146, gsm8k/147, gsm8k/148, gsm8k/149, gsm8k/15, gsm8k/150, gsm8k/152, gsm8k/153, gsm8k/154, gsm8k/155, gsm8k/156, gsm8k/157, gsm8k/158, gsm8k/159, gsm8k/16, gsm8k/160, gsm8k/161, gsm8k/162, gsm8k/163, gsm8k/164, gsm8k/165, gsm8k/166, gsm8k/167, gsm8k/168, gsm8k/169, gsm8k/17, gsm8k/170, gsm8k/171, gsm8k/172, gsm8k/173, gsm8k/174, gsm8k/177, gsm8k/178, gsm8k/179, gsm8k/18, gsm8k/180, gsm8k/181, gsm8k/182, gsm8k/183, gsm8k/184, gsm8k/185, gsm8k/186, gsm8k/187, gsm8k/188, gsm8k/189, gsm8k/19, gsm8k/190, gsm8k/191, gsm8k/192, gsm8k/193, gsm8k/195, gsm8k/196, gsm8k/197, gsm8k/198, gsm8k/199, gsm8k/2, gsm8k/200, gsm8k/201, gsm8k/202, gsm8k/203, gsm8k/204, gsm8k/205, gsm8k/206, gsm8k/207, gsm8k/208, gsm8k/209, gsm8k/21, gsm8k/211, gsm8k/212, gsm8k/213, gsm8k/214, gsm8k/215, gsm8k/216, gsm8k/217, gsm8k/218, gsm8k/219, gsm8k/22, gsm8k/220, gsm8k/222, gsm8k/223, gsm8k/224, gsm8k/225, gsm8k/226, gsm8k/227, gsm8k/228, gsm8k/229, gsm8k/23, gsm8k/230, gsm8k/231, gsm8k/232, gsm8k/233, gsm8k/234, gsm8k/236, gsm8k/237, gsm8k/238, gsm8k/239, gsm8k/24, gsm8k/240, gsm8k/241, gsm8k/242, gsm8k/243, gsm8k/244, gsm8k/245, gsm8k/246, gsm8k/247, gsm8k/248, gsm8k/249, gsm8k/25, gsm8k/250, gsm8k/251, gsm8k/252, gsm8k/253, gsm8k/254, gsm8k/255, gsm8k/256, gsm8k/257, gsm8k/258, gsm8k/259, gsm8k/26, gsm8k/260, gsm8k/261, gsm8k/262, gsm8k/263, gsm8k/264, gsm8k/265, gsm8k/266, gsm8k/267, gsm8k/268, gsm8k/269, gsm8k/27, gsm8k/270, gsm8k/271, gsm8k/272, gsm8k/273, gsm8k/274, gsm8k/275, gsm8k/276, gsm8k/277, gsm8k/278, gsm8k/28, gsm8k/280, gsm8k/281, gsm8k/282, gsm8k/283, gsm8k/284, gsm8k/285, gsm8k/286, gsm8k/287, gsm8k/288, gsm8k/289, gsm8k/29, gsm8k/290, gsm8k/291, gsm8k/292, gsm8k/293, gsm8k/295, gsm8k/296, gsm8k/297, gsm8k/298, gsm8k/299, gsm8k/3, gsm8k/30, gsm8k/300, gsm8k/301, gsm8k/302, gsm8k/303, gsm8k/304, gsm8k/305, gsm8k/307, gsm8k/308, gsm8k/309, gsm8k/31, gsm8k/310, gsm8k/311, gsm8k/312, gsm8k/313, gsm8k/314, gsm8k/315, gsm8k/316, gsm8k/317, gsm8k/318, gsm8k/319, gsm8k/32, gsm8k/320, gsm8k/322, gsm8k/323, gsm8k/324, gsm8k/325, gsm8k/326, gsm8k/327, gsm8k/328, gsm8k/329, gsm8k/33, gsm8k/330, gsm8k/331, gsm8k/332, gsm8k/333, gsm8k/334, gsm8k/335, gsm8k/336, gsm8k/338, gsm8k/339, gsm8k/34, gsm8k/340, gsm8k/341, gsm8k/342, gsm8k/343, gsm8k/344, gsm8k/345, gsm8k/346, gsm8k/347, gsm8k/348, gsm8k/349, gsm8k/35, gsm8k/352, gsm8k/353, gsm8k/354, gsm8k/355, gsm8k/356, gsm8k/357, gsm8k/358, gsm8k/359, gsm8k/36, gsm8k/360, gsm8k/361, gsm8k/362, gsm8k/363, gsm8k/364, gsm8k/365, gsm8k/366, gsm8k/367, gsm8k/368, gsm8k/369, gsm8k/37, gsm8k/370, gsm8k/371, gsm8k/372, gsm8k/373, gsm8k/374, gsm8k/375, gsm8k/376, gsm8k/377, gsm8k/379, gsm8k/380, gsm8k/381, gsm8k/382, gsm8k/383, gsm8k/384, gsm8k/385, gsm8k/386, gsm8k/387, gsm8k/388, gsm8k/389, gsm8k/39, gsm8k/390, gsm8k/391, gsm8k/392, gsm8k/393, gsm8k/394, gsm8k/395, gsm8k/396, gsm8k/397, gsm8k/398, gsm8k/399, gsm8k/40, gsm8k/400, gsm8k/401, gsm8k/402, gsm8k/403, gsm8k/404, gsm8k/405, gsm8k/406, gsm8k/409, gsm8k/41, gsm8k/410, gsm8k/411, gsm8k/412, gsm8k/413, gsm8k/414, gsm8k/415, gsm8k/416, gsm8k/417, gsm8k/418, gsm8k/42, gsm8k/420, gsm8k/421, gsm8k/422, gsm8k/423, gsm8k/424, gsm8k/425, gsm8k/426, gsm8k/427, gsm8k/428, gsm8k/429, gsm8k/43, gsm8k/430, gsm8k/431, gsm8k/432, gsm8k/433, gsm8k/434, gsm8k/436, gsm8k/437, gsm8k/439, gsm8k/440, gsm8k/441, gsm8k/442, gsm8k/443, gsm8k/444, gsm8k/445, gsm8k/446, gsm8k/447, gsm8k/448, gsm8k/449, gsm8k/45, gsm8k/450, gsm8k/451, gsm8k/452, gsm8k/453, gsm8k/454, gsm8k/455, gsm8k/456, gsm8k/457, gsm8k/458, gsm8k/46, gsm8k/460, gsm8k/461, gsm8k/462, gsm8k/463, gsm8k/464, gsm8k/465, gsm8k/466, gsm8k/467, gsm8k/468, gsm8k/469, gsm8k/47, gsm8k/470, gsm8k/471, gsm8k/472, gsm8k/474, gsm8k/475, gsm8k/476, gsm8k/477, gsm8k/478, gsm8k/479, gsm8k/48, gsm8k/480, gsm8k/481, gsm8k/482, gsm8k/483, gsm8k/484, gsm8k/485, gsm8k/486, gsm8k/487, gsm8k/488, gsm8k/489, gsm8k/49, gsm8k/490, gsm8k/491, gsm8k/492, gsm8k/493, gsm8k/494, gsm8k/495, gsm8k/496, gsm8k/497, gsm8k/498, gsm8k/5, gsm8k/50, gsm8k/500, gsm8k/501, gsm8k/502, gsm8k/503, gsm8k/504, gsm8k/505, gsm8k/506, gsm8k/508, gsm8k/51, gsm8k/510, gsm8k/511, gsm8k/512, gsm8k/513, gsm8k/514, gsm8k/515, gsm8k/516, gsm8k/517, gsm8k/518, gsm8k/519, gsm8k/52, gsm8k/520, gsm8k/521, gsm8k/522, gsm8k/523, gsm8k/524, gsm8k/526, gsm8k/527, gsm8k/528, gsm8k/529, gsm8k/530, gsm8k/531, gsm8k/532, gsm8k/533, gsm8k/534, gsm8k/535, gsm8k/536, gsm8k/537, gsm8k/539, gsm8k/54, gsm8k/540, gsm8k/541, gsm8k/542, gsm8k/543, gsm8k/544, gsm8k/545, gsm8k/546, gsm8k/547, gsm8k/549, gsm8k/55, gsm8k/550, gsm8k/551, gsm8k/552, gsm8k/553, gsm8k/554, gsm8k/556, gsm8k/557, gsm8k/558, gsm8k/559, gsm8k/56, gsm8k/560, gsm8k/561, gsm8k/562, gsm8k/564, gsm8k/565, gsm8k/567, gsm8k/568, gsm8k/57, gsm8k/570, gsm8k/571, gsm8k/573, gsm8k/574, gsm8k/575, gsm8k/576, gsm8k/577, gsm8k/578, gsm8k/579, gsm8k/58, gsm8k/580, gsm8k/581, gsm8k/582, gsm8k/583, gsm8k/585, gsm8k/586, gsm8k/587, gsm8k/589, gsm8k/59, gsm8k/590, gsm8k/591, gsm8k/593, gsm8k/594, gsm8k/595, gsm8k/596, gsm8k/597, gsm8k/598, gsm8k/599, gsm8k/6, gsm8k/60, gsm8k/600, gsm8k/601, gsm8k/602, gsm8k/603, gsm8k/604, gsm8k/605, gsm8k/606, gsm8k/607, gsm8k/609, gsm8k/61, gsm8k/610, gsm8k/611, gsm8k/612, gsm8k/613, gsm8k/614, gsm8k/615, gsm8k/616, gsm8k/617, gsm8k/618, gsm8k/619, gsm8k/62, gsm8k/620, gsm8k/621, gsm8k/622, gsm8k/623, gsm8k/624, gsm8k/625, gsm8k/626, gsm8k/627, gsm8k/629, gsm8k/63, gsm8k/630, gsm8k/631, gsm8k/632, gsm8k/633, gsm8k/634, gsm8k/635, gsm8k/636, gsm8k/637, gsm8k/638, gsm8k/639, gsm8k/64, gsm8k/640, gsm8k/641, gsm8k/642, gsm8k/643, gsm8k/644, gsm8k/645, gsm8k/646, gsm8k/647, gsm8k/648, gsm8k/649, gsm8k/65, gsm8k/650, gsm8k/651, gsm8k/652, gsm8k/653, gsm8k/654, gsm8k/657, gsm8k/658, gsm8k/659, gsm8k/66, gsm8k/660, gsm8k/661, gsm8k/662, gsm8k/663, gsm8k/664, gsm8k/665, gsm8k/667, gsm8k/668, gsm8k/669, gsm8k/67, gsm8k/671, gsm8k/672, gsm8k/673, gsm8k/674, gsm8k/675, gsm8k/676, gsm8k/677, gsm8k/678, gsm8k/679, gsm8k/68, gsm8k/680, gsm8k/681, gsm8k/682, gsm8k/683, gsm8k/684, gsm8k/685, gsm8k/686, gsm8k/687, gsm8k/688, gsm8k/689, gsm8k/69, gsm8k/690, gsm8k/691, gsm8k/692, gsm8k/693, gsm8k/694, gsm8k/695, gsm8k/696, gsm8k/697, gsm8k/698, gsm8k/699, gsm8k/7, gsm8k/70, gsm8k/700, gsm8k/701, gsm8k/702, gsm8k/703, gsm8k/704, gsm8k/705, gsm8k/706, gsm8k/707, gsm8k/708, gsm8k/709, gsm8k/71, gsm8k/710, gsm8k/711, gsm8k/712, gsm8k/713, gsm8k/714, gsm8k/715, gsm8k/716, gsm8k/717, gsm8k/718, gsm8k/719, gsm8k/72, gsm8k/721, gsm8k/722, gsm8k/723, gsm8k/724, gsm8k/725, gsm8k/727, gsm8k/728, gsm8k/729, gsm8k/73, gsm8k/732, gsm8k/733, gsm8k/734, gsm8k/736, gsm8k/737, gsm8k/738, gsm8k/739, gsm8k/74, gsm8k/740, gsm8k/741, gsm8k/742, gsm8k/743, gsm8k/744, gsm8k/745, gsm8k/746, gsm8k/747, gsm8k/748, gsm8k/749, gsm8k/75, gsm8k/750, gsm8k/752, gsm8k/753, gsm8k/754, gsm8k/755, gsm8k/756, gsm8k/757, gsm8k/758, gsm8k/759, gsm8k/76, gsm8k/760, gsm8k/761, gsm8k/762, gsm8k/764, gsm8k/765, gsm8k/766, gsm8k/767, gsm8k/768, gsm8k/769, gsm8k/771, gsm8k/772, gsm8k/775, gsm8k/776, gsm8k/777, gsm8k/778, gsm8k/779, gsm8k/78, gsm8k/780, gsm8k/781, gsm8k/782, gsm8k/783, gsm8k/784, gsm8k/785, gsm8k/786, gsm8k/787, gsm8k/789, gsm8k/79, gsm8k/790, gsm8k/791, gsm8k/792, gsm8k/793, gsm8k/794, gsm8k/795, gsm8k/796, gsm8k/797, gsm8k/799, gsm8k/8, gsm8k/80, gsm8k/800, gsm8k/801, gsm8k/802, gsm8k/803, gsm8k/805, gsm8k/806, gsm8k/807, gsm8k/809, gsm8k/81, gsm8k/810, gsm8k/811, gsm8k/812, gsm8k/813, gsm8k/814, gsm8k/815, gsm8k/816, gsm8k/817, gsm8k/818, gsm8k/819, gsm8k/82, gsm8k/820, gsm8k/821, gsm8k/822, gsm8k/824, gsm8k/825, gsm8k/826, gsm8k/827, gsm8k/828, gsm8k/829, gsm8k/83, gsm8k/830, gsm8k/831, gsm8k/832, gsm8k/833, gsm8k/834, gsm8k/835, gsm8k/836, gsm8k/837, gsm8k/838, gsm8k/839, gsm8k/840, gsm8k/841, gsm8k/842, gsm8k/843, gsm8k/844, gsm8k/845, gsm8k/846, gsm8k/847, gsm8k/848, gsm8k/849, gsm8k/85, gsm8k/850, gsm8k/851, gsm8k/852, gsm8k/853, gsm8k/854, gsm8k/855, gsm8k/856, gsm8k/857, gsm8k/858, gsm8k/859, gsm8k/86, gsm8k/860, gsm8k/861, gsm8k/862, gsm8k/863, gsm8k/864, gsm8k/865, gsm8k/866, gsm8k/867, gsm8k/868, gsm8k/869, gsm8k/87, gsm8k/870, gsm8k/871, gsm8k/872, gsm8k/873, gsm8k/874, gsm8k/875, gsm8k/876, gsm8k/877, gsm8k/878, gsm8k/879, gsm8k/88, gsm8k/880, gsm8k/881, gsm8k/882, gsm8k/883, gsm8k/884, gsm8k/885, gsm8k/886, gsm8k/887, gsm8k/889, gsm8k/89, gsm8k/890, gsm8k/891, gsm8k/892, gsm8k/893, gsm8k/894, gsm8k/895, gsm8k/896, gsm8k/897, gsm8k/898, gsm8k/899, gsm8k/9, gsm8k/90, gsm8k/900, gsm8k/901, gsm8k/902, gsm8k/903, gsm8k/904, gsm8k/905, gsm8k/906, gsm8k/907, gsm8k/908, gsm8k/909, gsm8k/91, gsm8k/910, gsm8k/911, gsm8k/912, gsm8k/913, gsm8k/914, gsm8k/915, gsm8k/916, gsm8k/917, gsm8k/918, gsm8k/919, gsm8k/920, gsm8k/921, gsm8k/922, gsm8k/923, gsm8k/924, gsm8k/925, gsm8k/927, gsm8k/928, gsm8k/929, gsm8k/93, gsm8k/930, gsm8k/931, gsm8k/932, gsm8k/933, gsm8k/934, gsm8k/935, gsm8k/936, gsm8k/937, gsm8k/938, gsm8k/939, gsm8k/94, gsm8k/940, gsm8k/941, gsm8k/942, gsm8k/943, gsm8k/944, gsm8k/945, gsm8k/946, gsm8k/947, gsm8k/948, gsm8k/95, gsm8k/950, gsm8k/951, gsm8k/952, gsm8k/953, gsm8k/954, gsm8k/957, gsm8k/958, gsm8k/959, gsm8k/96, gsm8k/960, gsm8k/961, gsm8k/962, gsm8k/963, gsm8k/964, gsm8k/965, gsm8k/966, gsm8k/967, gsm8k/968, gsm8k/969, gsm8k/97, gsm8k/970, gsm8k/972, gsm8k/973, gsm8k/974, gsm8k/975, gsm8k/976, gsm8k/977, gsm8k/978, gsm8k/979, gsm8k/980, gsm8k/981, gsm8k/982, gsm8k/983, gsm8k/984, gsm8k/985, gsm8k/986, gsm8k/987, gsm8k/988, gsm8k/989, gsm8k/99, gsm8k/990, gsm8k/991, gsm8k/992, gsm8k/993, gsm8k/994, gsm8k/995, gsm8k/996, gsm8k/997, gsm8k/998, gsm8k/999
| example_link | model | min_pass1_of_model |
|---|---|---|
| gsm8k/1118 | seed30841 | 0.019 |
| gsm8k/1114 | seed30841 | 0.019 |
| gsm8k/1234 | seed30841 | 0.019 |
| gsm8k/92 | seed30841 | 0.019 |
| gsm8k/730 | seed30841 | 0.019 |
| gsm8k/774 | seed30841 | 0.019 |
| gsm8k/808 | seed30841 | 0.019 |
| gsm8k/84 | seed30841 | 0.019 |
| gsm8k/38 | seed16385 | 0.017 |
| gsm8k/1304 | seed16385 | 0.017 |
| gsm8k/525 | seed16385 | 0.017 |
| gsm8k/773 | seed16385 | 0.017 |
| gsm8k/572 | seed16385 | 0.017 |
| gsm8k/507 | seed16385 | 0.017 |
| gsm8k/1036 | seed16385 | 0.017 |
| gsm8k/1216 | seed16385 | 0.017 |
| gsm8k/1038 | seed6529 | 0.017 |
| gsm8k/735 | seed6529 | 0.017 |
| gsm8k/798 | seed6529 | 0.017 |
| gsm8k/351 | seed6529 | 0.017 |
| gsm8k/408 | seed6529 | 0.017 |
| gsm8k/956 | seed17495 | 0.016 |
| gsm8k/419 | seed17495 | 0.016 |
| gsm8k/804 | seed24734 | 0.016 |
| gsm8k/888 | seed24734 | 0.016 |
| gsm8k/1173 | seed24734 | 0.016 |
| gsm8k/1037 | seed24734 | 0.016 |
| gsm8k/53 | seed17495 | 0.016 |
| gsm8k/279 | seed24734 | 0.016 |
| gsm8k/175 | seed24734 | 0.016 |
| gsm8k/20 | seed24734 | 0.016 |
| gsm8k/1241 | seed17495 | 0.016 |
| gsm8k/1185 | seed17495 | 0.016 |
| gsm8k/438 | seed17495 | 0.016 |
| gsm8k/435 | seed6115 | 0.015 |
| gsm8k/763 | seed6115 | 0.015 |
| gsm8k/98 | seed6115 | 0.015 |
| gsm8k/955 | seed6115 | 0.015 |
| gsm8k/949 | seed149 | 0.014 |
| gsm8k/350 | seed149 | 0.014 |
| gsm8k/592 | seed24762 | 0.013 |
| gsm8k/823 | seed24762 | 0.013 |
| gsm8k/306 | seed24762 | 0.013 |
| gsm8k/176 | seed24762 | 0.013 |
| gsm8k/770 | seed24762 | 0.013 |
| gsm8k/971 | seed24762 | 0.013 |
| gsm8k/337 | seed14309 | 0.011 |
| gsm8k/151 | seed27712 | 0.011 |
| gsm8k/1199 | seed27712 | 0.011 |
| gsm8k/1214 | seed14309 | 0.011 |
| gsm8k/294 | seed27712 | 0.011 |
| gsm8k/751 | seed14309 | 0.011 |
| gsm8k/555 | seed27712 | 0.011 |
These are 10 problems with the lowest correlation with the overall evaluation (i.e. better models tend to do worse on these. )
| example_link | pass1_of_ex | tau |
|---|---|---|
| gsm8k/221 | 0.800 | -0.419 |
| gsm8k/1199 | 0.100 | -0.407 |
| gsm8k/151 | 0.100 | -0.407 |
| gsm8k/1214 | 0.100 | -0.407 |
| gsm8k/294 | 0.100 | -0.407 |
| gsm8k/337 | 0.100 | -0.407 |
| gsm8k/555 | 0.100 | -0.407 |
| gsm8k/751 | 0.100 | -0.407 |
| gsm8k/726 | 0.700 | -0.366 |
| gsm8k/378 | 0.400 | -0.342 |
Histogram of problems by the accuracy on each problem.
Histogram of problems by the minimum win rate to solve each problem.