arc_challenge_step02500: by examples

Results Paper Code


Not solved by any model

There are 708 examples not solved by any model. Solving some of these can be a good signal that your model is indeed better than leading models if these are good problems.
arc_challenge/0, arc_challenge/1, arc_challenge/10, arc_challenge/1000, arc_challenge/1003, arc_challenge/1008, arc_challenge/1009, arc_challenge/1012, arc_challenge/1013, arc_challenge/1016, arc_challenge/1017, arc_challenge/1018, arc_challenge/1019, arc_challenge/1020, arc_challenge/1021, arc_challenge/1022, arc_challenge/1023, arc_challenge/1024, arc_challenge/1025, arc_challenge/1026, arc_challenge/1027, arc_challenge/1028, arc_challenge/1029, arc_challenge/1030, arc_challenge/1031, arc_challenge/1033, arc_challenge/1034, arc_challenge/1035, arc_challenge/1041, arc_challenge/1042, arc_challenge/1043, arc_challenge/1044, arc_challenge/1045, arc_challenge/1046, arc_challenge/1047, arc_challenge/1049, arc_challenge/105, arc_challenge/1050, arc_challenge/1051, arc_challenge/1058, arc_challenge/1060, arc_challenge/1062, arc_challenge/1064, arc_challenge/1065, arc_challenge/1066, arc_challenge/1068, arc_challenge/107, arc_challenge/1070, arc_challenge/1071, arc_challenge/1074, arc_challenge/1075, arc_challenge/1076, arc_challenge/1077, arc_challenge/1078, arc_challenge/1079, arc_challenge/1081, arc_challenge/1082, arc_challenge/1085, arc_challenge/1086, arc_challenge/1088, arc_challenge/109, arc_challenge/1091, arc_challenge/1093, arc_challenge/1094, arc_challenge/1096, arc_challenge/1097, arc_challenge/1101, arc_challenge/1102, arc_challenge/1103, arc_challenge/1104, arc_challenge/1106, arc_challenge/1107, arc_challenge/1109, arc_challenge/111, arc_challenge/1110, arc_challenge/1111, arc_challenge/1112, arc_challenge/1114, arc_challenge/1115, arc_challenge/1116, arc_challenge/112, arc_challenge/1122, arc_challenge/1123, arc_challenge/1125, arc_challenge/1126, arc_challenge/1129, arc_challenge/113, arc_challenge/1130, arc_challenge/1131, arc_challenge/1132, arc_challenge/1133, arc_challenge/1137, arc_challenge/1138, arc_challenge/1139, arc_challenge/1140, arc_challenge/1142, arc_challenge/1143, arc_challenge/1144, arc_challenge/1146, arc_challenge/1147, arc_challenge/1152, arc_challenge/1156, arc_challenge/1157, arc_challenge/1160, arc_challenge/1161, arc_challenge/1164, arc_challenge/119, arc_challenge/12, arc_challenge/121, arc_challenge/123, arc_challenge/125, arc_challenge/126, arc_challenge/127, arc_challenge/128, arc_challenge/13, arc_challenge/130, arc_challenge/132, arc_challenge/133, arc_challenge/134, arc_challenge/136, arc_challenge/137, arc_challenge/138, arc_challenge/14, arc_challenge/140, arc_challenge/141, arc_challenge/142, arc_challenge/144, arc_challenge/145, arc_challenge/147, arc_challenge/148, arc_challenge/149, arc_challenge/151, arc_challenge/153, arc_challenge/154, arc_challenge/155, arc_challenge/156, arc_challenge/157, arc_challenge/158, arc_challenge/159, arc_challenge/160, arc_challenge/161, arc_challenge/162, arc_challenge/163, arc_challenge/165, arc_challenge/166, arc_challenge/167, arc_challenge/169, arc_challenge/17, arc_challenge/170, arc_challenge/171, arc_challenge/172, arc_challenge/173, arc_challenge/174, arc_challenge/176, arc_challenge/177, arc_challenge/179, arc_challenge/18, arc_challenge/181, arc_challenge/182, arc_challenge/185, arc_challenge/186, arc_challenge/19, arc_challenge/191, arc_challenge/193, arc_challenge/195, arc_challenge/197, arc_challenge/198, arc_challenge/2, arc_challenge/20, arc_challenge/201, arc_challenge/202, arc_challenge/204, arc_challenge/208, arc_challenge/209, arc_challenge/211, arc_challenge/212, arc_challenge/215, arc_challenge/217, arc_challenge/218, arc_challenge/219, arc_challenge/22, arc_challenge/220, arc_challenge/221, arc_challenge/222, arc_challenge/227, arc_challenge/229, arc_challenge/230, arc_challenge/232, arc_challenge/233, arc_challenge/235, arc_challenge/236, arc_challenge/237, arc_challenge/240, arc_challenge/241, arc_challenge/242, arc_challenge/243, arc_challenge/244, arc_challenge/245, arc_challenge/246, arc_challenge/247, arc_challenge/248, arc_challenge/249, arc_challenge/25, arc_challenge/250, arc_challenge/251, arc_challenge/252, arc_challenge/253, arc_challenge/257, arc_challenge/259, arc_challenge/26, arc_challenge/262, arc_challenge/263, arc_challenge/265, arc_challenge/266, arc_challenge/268, arc_challenge/269, arc_challenge/27, arc_challenge/270, arc_challenge/271, arc_challenge/276, arc_challenge/278, arc_challenge/279, arc_challenge/28, arc_challenge/281, arc_challenge/282, arc_challenge/284, arc_challenge/285, arc_challenge/286, arc_challenge/287, arc_challenge/291, arc_challenge/292, arc_challenge/293, arc_challenge/297, arc_challenge/298, arc_challenge/30, arc_challenge/302, arc_challenge/303, arc_challenge/306, arc_challenge/307, arc_challenge/308, arc_challenge/310, arc_challenge/311, arc_challenge/312, arc_challenge/313, arc_challenge/314, arc_challenge/315, arc_challenge/316, arc_challenge/317, arc_challenge/318, arc_challenge/320, arc_challenge/322, arc_challenge/323, arc_challenge/324, arc_challenge/326, arc_challenge/327, arc_challenge/328, arc_challenge/329, arc_challenge/333, arc_challenge/334, arc_challenge/336, arc_challenge/337, arc_challenge/34, arc_challenge/341, arc_challenge/344, arc_challenge/345, arc_challenge/347, arc_challenge/348, arc_challenge/349, arc_challenge/35, arc_challenge/350, arc_challenge/353, arc_challenge/354, arc_challenge/357, arc_challenge/358, arc_challenge/359, arc_challenge/36, arc_challenge/360, arc_challenge/361, arc_challenge/362, arc_challenge/364, arc_challenge/365, arc_challenge/366, arc_challenge/368, arc_challenge/369, arc_challenge/37, arc_challenge/370, arc_challenge/371, arc_challenge/373, arc_challenge/374, arc_challenge/376, arc_challenge/377, arc_challenge/379, arc_challenge/380, arc_challenge/381, arc_challenge/382, arc_challenge/383, arc_challenge/385, arc_challenge/387, arc_challenge/389, arc_challenge/39, arc_challenge/390, arc_challenge/391, arc_challenge/392, arc_challenge/394, arc_challenge/397, arc_challenge/398, arc_challenge/399, arc_challenge/4, arc_challenge/400, arc_challenge/401, arc_challenge/402, arc_challenge/403, arc_challenge/404, arc_challenge/405, arc_challenge/406, arc_challenge/41, arc_challenge/411, arc_challenge/413, arc_challenge/414, arc_challenge/417, arc_challenge/418, arc_challenge/419, arc_challenge/42, arc_challenge/420, arc_challenge/421, arc_challenge/423, arc_challenge/424, arc_challenge/425, arc_challenge/426, arc_challenge/427, arc_challenge/432, arc_challenge/433, arc_challenge/434, arc_challenge/435, arc_challenge/436, arc_challenge/438, arc_challenge/439, arc_challenge/440, arc_challenge/441, arc_challenge/445, arc_challenge/446, arc_challenge/447, arc_challenge/448, arc_challenge/45, arc_challenge/450, arc_challenge/457, arc_challenge/458, arc_challenge/459, arc_challenge/461, arc_challenge/462, arc_challenge/463, arc_challenge/464, arc_challenge/466, arc_challenge/468, arc_challenge/47, arc_challenge/470, arc_challenge/472, arc_challenge/474, arc_challenge/475, arc_challenge/476, arc_challenge/478, arc_challenge/479, arc_challenge/48, arc_challenge/481, arc_challenge/482, arc_challenge/483, arc_challenge/487, arc_challenge/488, arc_challenge/489, arc_challenge/49, arc_challenge/490, arc_challenge/492, arc_challenge/495, arc_challenge/496, arc_challenge/497, arc_challenge/5, arc_challenge/50, arc_challenge/500, arc_challenge/501, arc_challenge/505, arc_challenge/506, arc_challenge/507, arc_challenge/508, arc_challenge/509, arc_challenge/51, arc_challenge/513, arc_challenge/515, arc_challenge/517, arc_challenge/518, arc_challenge/520, arc_challenge/521, arc_challenge/522, arc_challenge/523, arc_challenge/524, arc_challenge/526, arc_challenge/528, arc_challenge/53, arc_challenge/530, arc_challenge/531, arc_challenge/532, arc_challenge/533, arc_challenge/535, arc_challenge/536, arc_challenge/54, arc_challenge/540, arc_challenge/542, arc_challenge/543, arc_challenge/544, arc_challenge/545, arc_challenge/547, arc_challenge/548, arc_challenge/549, arc_challenge/55, arc_challenge/551, arc_challenge/552, arc_challenge/555, arc_challenge/556, arc_challenge/559, arc_challenge/56, arc_challenge/560, arc_challenge/561, arc_challenge/562, arc_challenge/564, arc_challenge/569, arc_challenge/57, arc_challenge/570, arc_challenge/571, arc_challenge/573, arc_challenge/574, arc_challenge/576, arc_challenge/577, arc_challenge/579, arc_challenge/580, arc_challenge/581, arc_challenge/583, arc_challenge/584, arc_challenge/585, arc_challenge/587, arc_challenge/588, arc_challenge/589, arc_challenge/59, arc_challenge/591, arc_challenge/592, arc_challenge/593, arc_challenge/594, arc_challenge/597, arc_challenge/598, arc_challenge/599, arc_challenge/6, arc_challenge/602, arc_challenge/604, arc_challenge/607, arc_challenge/609, arc_challenge/610, arc_challenge/611, arc_challenge/612, arc_challenge/613, arc_challenge/614, arc_challenge/615, arc_challenge/617, arc_challenge/619, arc_challenge/620, arc_challenge/622, arc_challenge/623, arc_challenge/624, arc_challenge/626, arc_challenge/627, arc_challenge/628, arc_challenge/63, arc_challenge/630, arc_challenge/632, arc_challenge/635, arc_challenge/637, arc_challenge/638, arc_challenge/64, arc_challenge/643, arc_challenge/644, arc_challenge/649, arc_challenge/65, arc_challenge/650, arc_challenge/651, arc_challenge/652, arc_challenge/654, arc_challenge/655, arc_challenge/659, arc_challenge/66, arc_challenge/660, arc_challenge/662, arc_challenge/664, arc_challenge/667, arc_challenge/668, arc_challenge/67, arc_challenge/670, arc_challenge/672, arc_challenge/674, arc_challenge/677, arc_challenge/678, arc_challenge/679, arc_challenge/681, arc_challenge/682, arc_challenge/683, arc_challenge/684, arc_challenge/685, arc_challenge/687, arc_challenge/688, arc_challenge/69, arc_challenge/690, arc_challenge/691, arc_challenge/693, arc_challenge/694, arc_challenge/695, arc_challenge/699, arc_challenge/702, arc_challenge/703, arc_challenge/704, arc_challenge/706, arc_challenge/708, arc_challenge/709, arc_challenge/71, arc_challenge/710, arc_challenge/711, arc_challenge/713, arc_challenge/714, arc_challenge/715, arc_challenge/716, arc_challenge/717, arc_challenge/718, arc_challenge/719, arc_challenge/72, arc_challenge/720, arc_challenge/721, arc_challenge/722, arc_challenge/723, arc_challenge/726, arc_challenge/727, arc_challenge/728, arc_challenge/730, arc_challenge/731, arc_challenge/732, arc_challenge/734, arc_challenge/736, arc_challenge/74, arc_challenge/740, arc_challenge/742, arc_challenge/743, arc_challenge/744, arc_challenge/746, arc_challenge/747, arc_challenge/748, arc_challenge/75, arc_challenge/750, arc_challenge/752, arc_challenge/753, arc_challenge/754, arc_challenge/755, arc_challenge/756, arc_challenge/757, arc_challenge/758, arc_challenge/759, arc_challenge/76, arc_challenge/760, arc_challenge/764, arc_challenge/765, arc_challenge/769, arc_challenge/770, arc_challenge/772, arc_challenge/773, arc_challenge/774, arc_challenge/776, arc_challenge/777, arc_challenge/778, arc_challenge/78, arc_challenge/783, arc_challenge/784, arc_challenge/786, arc_challenge/789, arc_challenge/79, arc_challenge/790, arc_challenge/791, arc_challenge/792, arc_challenge/793, arc_challenge/795, arc_challenge/796, arc_challenge/797, arc_challenge/798, arc_challenge/799, arc_challenge/8, arc_challenge/80, arc_challenge/800, arc_challenge/801, arc_challenge/802, arc_challenge/803, arc_challenge/805, arc_challenge/806, arc_challenge/807, arc_challenge/808, arc_challenge/81, arc_challenge/810, arc_challenge/812, arc_challenge/813, arc_challenge/815, arc_challenge/816, arc_challenge/817, arc_challenge/818, arc_challenge/819, arc_challenge/82, arc_challenge/820, arc_challenge/822, arc_challenge/824, arc_challenge/826, arc_challenge/828, arc_challenge/829, arc_challenge/830, arc_challenge/833, arc_challenge/834, arc_challenge/835, arc_challenge/838, arc_challenge/839, arc_challenge/840, arc_challenge/841, arc_challenge/844, arc_challenge/847, arc_challenge/848, arc_challenge/849, arc_challenge/850, arc_challenge/851, arc_challenge/852, arc_challenge/857, arc_challenge/860, arc_challenge/861, arc_challenge/865, arc_challenge/868, arc_challenge/869, arc_challenge/87, arc_challenge/870, arc_challenge/872, arc_challenge/873, arc_challenge/874, arc_challenge/876, arc_challenge/877, arc_challenge/878, arc_challenge/879, arc_challenge/88, arc_challenge/880, arc_challenge/881, arc_challenge/882, arc_challenge/885, arc_challenge/886, arc_challenge/89, arc_challenge/890, arc_challenge/891, arc_challenge/892, arc_challenge/895, arc_challenge/897, arc_challenge/9, arc_challenge/901, arc_challenge/902, arc_challenge/903, arc_challenge/905, arc_challenge/908, arc_challenge/912, arc_challenge/914, arc_challenge/915, arc_challenge/917, arc_challenge/919, arc_challenge/92, arc_challenge/921, arc_challenge/922, arc_challenge/923, arc_challenge/925, arc_challenge/926, arc_challenge/93, arc_challenge/932, arc_challenge/933, arc_challenge/934, arc_challenge/939, arc_challenge/943, arc_challenge/949, arc_challenge/953, arc_challenge/955, arc_challenge/957, arc_challenge/958, arc_challenge/959, arc_challenge/960, arc_challenge/961, arc_challenge/962, arc_challenge/963, arc_challenge/964, arc_challenge/965, arc_challenge/967, arc_challenge/971, arc_challenge/972, arc_challenge/973, arc_challenge/974, arc_challenge/976, arc_challenge/978, arc_challenge/979, arc_challenge/980, arc_challenge/982, arc_challenge/983, arc_challenge/985, arc_challenge/986, arc_challenge/989, arc_challenge/990, arc_challenge/991, arc_challenge/992, arc_challenge/996, arc_challenge/999

Problems solved by 1 model only

example_link model min_pass1_of_model
arc_challenge/118 seed149 0.254
arc_challenge/346 seed149 0.254
arc_challenge/319 seed149 0.254
arc_challenge/510 seed149 0.254
arc_challenge/519 seed149 0.254
arc_challenge/294 seed149 0.254
arc_challenge/60 seed149 0.254
arc_challenge/58 seed149 0.254
arc_challenge/821 seed149 0.254
arc_challenge/855 seed149 0.254
arc_challenge/946 seed149 0.254
arc_challenge/916 seed149 0.254
arc_challenge/621 seed149 0.254
arc_challenge/299 seed30841 0.252
arc_challenge/184 seed30841 0.252
arc_challenge/1056 seed30841 0.252
arc_challenge/725 seed30841 0.252
arc_challenge/325 seed30841 0.252
arc_challenge/1067 seed30841 0.252
arc_challenge/864 seed30841 0.252
arc_challenge/814 seed30841 0.252
arc_challenge/920 seed30841 0.252
arc_challenge/893 seed6115 0.242
arc_challenge/629 seed6115 0.242
arc_challenge/1149 seed6115 0.242
arc_challenge/129 seed6115 0.242
arc_challenge/1038 seed6115 0.242
arc_challenge/431 seed6115 0.242
arc_challenge/1037 seed6115 0.242
arc_challenge/378 seed27712 0.241
arc_challenge/948 seed27712 0.241
arc_challenge/618 seed27712 0.241
arc_challenge/729 seed6529 0.240
arc_challenge/846 seed6529 0.240
arc_challenge/375 seed6529 0.240
arc_challenge/1119 seed6529 0.240
arc_challenge/395 seed6529 0.240
arc_challenge/29 seed6529 0.240
arc_challenge/84 seed6529 0.240
arc_challenge/883 seed16385 0.239
arc_challenge/196 seed16385 0.239
arc_challenge/254 seed24762 0.239
arc_challenge/183 seed16385 0.239
arc_challenge/896 seed24762 0.239
arc_challenge/733 seed24762 0.239
arc_challenge/288 seed24762 0.239
arc_challenge/267 seed24762 0.239
arc_challenge/356 seed16385 0.239
arc_challenge/984 seed24762 0.239
arc_challenge/646 seed24762 0.239
arc_challenge/1057 seed24734 0.235
arc_challenge/539 seed24734 0.235
arc_challenge/415 seed24734 0.235
arc_challenge/899 seed24734 0.235
arc_challenge/993 seed17495 0.232
arc_challenge/367 seed17495 0.232
arc_challenge/977 seed17495 0.232
arc_challenge/781 seed17495 0.232
arc_challenge/653 seed17495 0.232
arc_challenge/696 seed17495 0.232
arc_challenge/1011 seed14309 0.227
arc_challenge/7 seed14309 0.227
arc_challenge/927 seed14309 0.227
arc_challenge/975 seed14309 0.227

Suspect problems

These are 10 problems with the lowest correlation with the overall evaluation (i.e. better models tend to do worse on these. )

example_link pass1_of_ex tau
arc_challenge/1108 0.700 -0.691
arc_challenge/1040 0.500 -0.663
arc_challenge/339 0.700 -0.625
arc_challenge/3 0.300 -0.592
arc_challenge/762 0.700 -0.559
arc_challenge/787 0.300 -0.526
arc_challenge/1053 0.500 -0.513
arc_challenge/527 0.400 -0.462
arc_challenge/473 0.300 -0.461
arc_challenge/600 0.900 -0.452

Histogram of accuracies

Histogram of problems by the accuracy on each problem.

Histogram of difficulties

Histogram of problems by the minimum win rate to solve each problem.