arc_challenge_step27500: by examples

Results Paper Code


Not solved by any model

There are 518 examples not solved by any model. Solving some of these can be a good signal that your model is indeed better than leading models if these are good problems.
arc_challenge/0, arc_challenge/1, arc_challenge/1001, arc_challenge/1003, arc_challenge/1011, arc_challenge/1012, arc_challenge/1013, arc_challenge/1019, arc_challenge/1020, arc_challenge/1021, arc_challenge/1022, arc_challenge/1024, arc_challenge/1025, arc_challenge/1026, arc_challenge/1030, arc_challenge/1031, arc_challenge/1034, arc_challenge/1038, arc_challenge/1042, arc_challenge/1045, arc_challenge/1047, arc_challenge/1049, arc_challenge/105, arc_challenge/1051, arc_challenge/1056, arc_challenge/1062, arc_challenge/1064, arc_challenge/1065, arc_challenge/1066, arc_challenge/107, arc_challenge/1070, arc_challenge/1071, arc_challenge/1074, arc_challenge/1075, arc_challenge/1077, arc_challenge/1079, arc_challenge/1083, arc_challenge/1085, arc_challenge/1086, arc_challenge/1088, arc_challenge/109, arc_challenge/1095, arc_challenge/1096, arc_challenge/1097, arc_challenge/1102, arc_challenge/1103, arc_challenge/1104, arc_challenge/1106, arc_challenge/1109, arc_challenge/111, arc_challenge/1110, arc_challenge/1111, arc_challenge/1114, arc_challenge/1115, arc_challenge/112, arc_challenge/1120, arc_challenge/1122, arc_challenge/1123, arc_challenge/1126, arc_challenge/1129, arc_challenge/113, arc_challenge/1132, arc_challenge/1133, arc_challenge/1135, arc_challenge/1137, arc_challenge/1138, arc_challenge/1139, arc_challenge/1140, arc_challenge/1143, arc_challenge/1146, arc_challenge/1147, arc_challenge/1148, arc_challenge/1152, arc_challenge/1156, arc_challenge/1157, arc_challenge/1160, arc_challenge/1161, arc_challenge/1164, arc_challenge/119, arc_challenge/123, arc_challenge/125, arc_challenge/127, arc_challenge/129, arc_challenge/13, arc_challenge/130, arc_challenge/133, arc_challenge/136, arc_challenge/138, arc_challenge/14, arc_challenge/142, arc_challenge/144, arc_challenge/149, arc_challenge/154, arc_challenge/156, arc_challenge/158, arc_challenge/159, arc_challenge/160, arc_challenge/162, arc_challenge/163, arc_challenge/167, arc_challenge/171, arc_challenge/172, arc_challenge/173, arc_challenge/174, arc_challenge/176, arc_challenge/177, arc_challenge/181, arc_challenge/182, arc_challenge/183, arc_challenge/185, arc_challenge/186, arc_challenge/19, arc_challenge/191, arc_challenge/193, arc_challenge/196, arc_challenge/2, arc_challenge/20, arc_challenge/202, arc_challenge/204, arc_challenge/208, arc_challenge/209, arc_challenge/212, arc_challenge/217, arc_challenge/218, arc_challenge/221, arc_challenge/222, arc_challenge/229, arc_challenge/230, arc_challenge/232, arc_challenge/233, arc_challenge/235, arc_challenge/240, arc_challenge/241, arc_challenge/242, arc_challenge/243, arc_challenge/247, arc_challenge/249, arc_challenge/250, arc_challenge/251, arc_challenge/252, arc_challenge/253, arc_challenge/254, arc_challenge/257, arc_challenge/259, arc_challenge/262, arc_challenge/263, arc_challenge/265, arc_challenge/266, arc_challenge/269, arc_challenge/27, arc_challenge/270, arc_challenge/271, arc_challenge/273, arc_challenge/276, arc_challenge/278, arc_challenge/279, arc_challenge/28, arc_challenge/281, arc_challenge/284, arc_challenge/285, arc_challenge/286, arc_challenge/287, arc_challenge/288, arc_challenge/289, arc_challenge/291, arc_challenge/293, arc_challenge/297, arc_challenge/298, arc_challenge/299, arc_challenge/303, arc_challenge/305, arc_challenge/306, arc_challenge/310, arc_challenge/311, arc_challenge/312, arc_challenge/313, arc_challenge/314, arc_challenge/315, arc_challenge/316, arc_challenge/317, arc_challenge/318, arc_challenge/324, arc_challenge/326, arc_challenge/327, arc_challenge/328, arc_challenge/329, arc_challenge/333, arc_challenge/336, arc_challenge/337, arc_challenge/34, arc_challenge/345, arc_challenge/347, arc_challenge/35, arc_challenge/352, arc_challenge/353, arc_challenge/354, arc_challenge/359, arc_challenge/360, arc_challenge/361, arc_challenge/362, arc_challenge/363, arc_challenge/364, arc_challenge/365, arc_challenge/366, arc_challenge/368, arc_challenge/369, arc_challenge/37, arc_challenge/370, arc_challenge/376, arc_challenge/379, arc_challenge/383, arc_challenge/389, arc_challenge/39, arc_challenge/390, arc_challenge/391, arc_challenge/392, arc_challenge/4, arc_challenge/400, arc_challenge/401, arc_challenge/402, arc_challenge/404, arc_challenge/405, arc_challenge/406, arc_challenge/408, arc_challenge/41, arc_challenge/411, arc_challenge/413, arc_challenge/417, arc_challenge/418, arc_challenge/419, arc_challenge/420, arc_challenge/423, arc_challenge/424, arc_challenge/426, arc_challenge/427, arc_challenge/432, arc_challenge/433, arc_challenge/435, arc_challenge/438, arc_challenge/440, arc_challenge/446, arc_challenge/447, arc_challenge/449, arc_challenge/45, arc_challenge/450, arc_challenge/456, arc_challenge/457, arc_challenge/458, arc_challenge/459, arc_challenge/461, arc_challenge/462, arc_challenge/464, arc_challenge/466, arc_challenge/468, arc_challenge/47, arc_challenge/474, arc_challenge/476, arc_challenge/478, arc_challenge/479, arc_challenge/483, arc_challenge/488, arc_challenge/489, arc_challenge/49, arc_challenge/492, arc_challenge/495, arc_challenge/497, arc_challenge/5, arc_challenge/50, arc_challenge/501, arc_challenge/506, arc_challenge/507, arc_challenge/508, arc_challenge/51, arc_challenge/511, arc_challenge/513, arc_challenge/517, arc_challenge/518, arc_challenge/520, arc_challenge/522, arc_challenge/523, arc_challenge/524, arc_challenge/526, arc_challenge/53, arc_challenge/531, arc_challenge/532, arc_challenge/535, arc_challenge/54, arc_challenge/542, arc_challenge/545, arc_challenge/547, arc_challenge/548, arc_challenge/549, arc_challenge/55, arc_challenge/551, arc_challenge/554, arc_challenge/556, arc_challenge/56, arc_challenge/560, arc_challenge/569, arc_challenge/570, arc_challenge/571, arc_challenge/574, arc_challenge/576, arc_challenge/577, arc_challenge/579, arc_challenge/580, arc_challenge/582, arc_challenge/583, arc_challenge/584, arc_challenge/587, arc_challenge/588, arc_challenge/589, arc_challenge/59, arc_challenge/592, arc_challenge/594, arc_challenge/597, arc_challenge/598, arc_challenge/599, arc_challenge/6, arc_challenge/602, arc_challenge/604, arc_challenge/609, arc_challenge/611, arc_challenge/612, arc_challenge/613, arc_challenge/614, arc_challenge/615, arc_challenge/619, arc_challenge/620, arc_challenge/621, arc_challenge/622, arc_challenge/623, arc_challenge/624, arc_challenge/625, arc_challenge/626, arc_challenge/627, arc_challenge/628, arc_challenge/629, arc_challenge/63, arc_challenge/632, arc_challenge/635, arc_challenge/637, arc_challenge/64, arc_challenge/644, arc_challenge/649, arc_challenge/650, arc_challenge/651, arc_challenge/652, arc_challenge/654, arc_challenge/655, arc_challenge/658, arc_challenge/66, arc_challenge/660, arc_challenge/662, arc_challenge/664, arc_challenge/67, arc_challenge/670, arc_challenge/672, arc_challenge/674, arc_challenge/677, arc_challenge/678, arc_challenge/679, arc_challenge/682, arc_challenge/683, arc_challenge/684, arc_challenge/685, arc_challenge/688, arc_challenge/69, arc_challenge/690, arc_challenge/691, arc_challenge/692, arc_challenge/693, arc_challenge/699, arc_challenge/702, arc_challenge/708, arc_challenge/709, arc_challenge/71, arc_challenge/711, arc_challenge/715, arc_challenge/716, arc_challenge/717, arc_challenge/718, arc_challenge/719, arc_challenge/72, arc_challenge/720, arc_challenge/723, arc_challenge/724, arc_challenge/726, arc_challenge/727, arc_challenge/728, arc_challenge/730, arc_challenge/732, arc_challenge/734, arc_challenge/737, arc_challenge/738, arc_challenge/740, arc_challenge/743, arc_challenge/746, arc_challenge/747, arc_challenge/75, arc_challenge/750, arc_challenge/751, arc_challenge/752, arc_challenge/753, arc_challenge/754, arc_challenge/755, arc_challenge/756, arc_challenge/757, arc_challenge/758, arc_challenge/759, arc_challenge/76, arc_challenge/760, arc_challenge/765, arc_challenge/77, arc_challenge/773, arc_challenge/774, arc_challenge/776, arc_challenge/777, arc_challenge/778, arc_challenge/78, arc_challenge/783, arc_challenge/789, arc_challenge/79, arc_challenge/790, arc_challenge/791, arc_challenge/792, arc_challenge/794, arc_challenge/795, arc_challenge/796, arc_challenge/797, arc_challenge/798, arc_challenge/8, arc_challenge/800, arc_challenge/802, arc_challenge/803, arc_challenge/805, arc_challenge/808, arc_challenge/81, arc_challenge/810, arc_challenge/812, arc_challenge/813, arc_challenge/815, arc_challenge/816, arc_challenge/818, arc_challenge/82, arc_challenge/820, arc_challenge/822, arc_challenge/826, arc_challenge/828, arc_challenge/830, arc_challenge/835, arc_challenge/838, arc_challenge/84, arc_challenge/840, arc_challenge/844, arc_challenge/852, arc_challenge/855, arc_challenge/857, arc_challenge/860, arc_challenge/861, arc_challenge/868, arc_challenge/873, arc_challenge/874, arc_challenge/875, arc_challenge/878, arc_challenge/879, arc_challenge/88, arc_challenge/880, arc_challenge/881, arc_challenge/882, arc_challenge/89, arc_challenge/890, arc_challenge/892, arc_challenge/897, arc_challenge/9, arc_challenge/901, arc_challenge/902, arc_challenge/903, arc_challenge/905, arc_challenge/908, arc_challenge/913, arc_challenge/914, arc_challenge/916, arc_challenge/919, arc_challenge/921, arc_challenge/922, arc_challenge/923, arc_challenge/926, arc_challenge/928, arc_challenge/93, arc_challenge/932, arc_challenge/934, arc_challenge/939, arc_challenge/942, arc_challenge/943, arc_challenge/952, arc_challenge/953, arc_challenge/957, arc_challenge/959, arc_challenge/960, arc_challenge/961, arc_challenge/962, arc_challenge/964, arc_challenge/965, arc_challenge/967, arc_challenge/971, arc_challenge/974, arc_challenge/977, arc_challenge/980, arc_challenge/981, arc_challenge/982, arc_challenge/983, arc_challenge/987, arc_challenge/989, arc_challenge/990, arc_challenge/991, arc_challenge/996, arc_challenge/999

Problems solved by 1 model only

example_link model min_pass1_of_model
arc_challenge/1008 seed149 0.385
arc_challenge/1033 seed149 0.385
arc_challenge/827 seed149 0.385
arc_challenge/307 seed149 0.385
arc_challenge/302 seed149 0.385
arc_challenge/437 seed149 0.385
arc_challenge/344 seed149 0.385
arc_challenge/215 seed149 0.385
arc_challenge/918 seed149 0.385
arc_challenge/559 seed149 0.385
arc_challenge/869 seed149 0.385
arc_challenge/973 seed24734 0.380
arc_challenge/533 seed24734 0.380
arc_challenge/872 seed24734 0.380
arc_challenge/414 seed24734 0.380
arc_challenge/537 seed24734 0.380
arc_challenge/157 seed24734 0.380
arc_challenge/1050 seed24734 0.380
arc_challenge/165 seed6115 0.379
arc_challenge/669 seed6115 0.379
arc_challenge/591 seed6115 0.379
arc_challenge/70 seed6115 0.379
arc_challenge/590 seed6115 0.379
arc_challenge/925 seed6115 0.379
arc_challenge/851 seed6115 0.379
arc_challenge/643 seed6115 0.379
arc_challenge/166 seed6115 0.379
arc_challenge/48 seed6115 0.379
arc_challenge/189 seed6115 0.379
arc_challenge/319 seed6529 0.363
arc_challenge/475 seed6529 0.363
arc_challenge/1130 seed6529 0.363
arc_challenge/1116 seed6529 0.363
arc_challenge/1078 seed6529 0.363
arc_challenge/1028 seed16385 0.362
arc_challenge/137 seed16385 0.362
arc_challenge/155 seed16385 0.362
arc_challenge/725 seed16385 0.362
arc_challenge/920 seed24762 0.361
arc_challenge/954 seed24762 0.361
arc_challenge/706 seed24762 0.361
arc_challenge/92 seed24762 0.361
arc_challenge/43 seed24762 0.361
arc_challenge/480 seed24762 0.361
arc_challenge/1043 seed24762 0.361
arc_challenge/1090 seed30841 0.360
arc_challenge/585 seed30841 0.360
arc_challenge/729 seed30841 0.360
arc_challenge/394 seed30841 0.360
arc_challenge/110 seed30841 0.360
arc_challenge/893 seed30841 0.360
arc_challenge/837 seed14309 0.357
arc_challenge/955 seed14309 0.357
arc_challenge/521 seed14309 0.357
arc_challenge/1041 seed14309 0.357
arc_challenge/121 seed14309 0.357
arc_challenge/441 seed14309 0.357
arc_challenge/593 seed14309 0.357
arc_challenge/985 seed17495 0.354
arc_challenge/341 seed17495 0.354
arc_challenge/742 seed17495 0.354
arc_challenge/80 seed17495 0.354
arc_challenge/425 seed27712 0.352
arc_challenge/681 seed27712 0.352
arc_challenge/65 seed27712 0.352
arc_challenge/849 seed27712 0.352

Suspect problems

These are 10 problems with the lowest correlation with the overall evaluation (i.e. better models tend to do worse on these. )

example_link pass1_of_ex tau
arc_challenge/371 0.600 -0.730
arc_challenge/1044 0.600 -0.730
arc_challenge/260 0.400 -0.609
arc_challenge/384 0.300 -0.553
arc_challenge/103 0.300 -0.553
arc_challenge/633 0.600 -0.548
arc_challenge/487 0.200 -0.522
arc_challenge/871 0.800 -0.522
arc_challenge/1155 0.800 -0.522
arc_challenge/10 0.200 -0.522

Histogram of accuracies

Histogram of problems by the accuracy on each problem.

Histogram of difficulties

Histogram of problems by the minimum win rate to solve each problem.