arc_challenge_step50000: by examples

Results Paper Code


Not solved by any model

There are 479 examples not solved by any model. Solving some of these can be a good signal that your model is indeed better than leading models if these are good problems.
arc_challenge/0, arc_challenge/10, arc_challenge/1003, arc_challenge/1011, arc_challenge/1012, arc_challenge/1013, arc_challenge/1016, arc_challenge/1020, arc_challenge/1021, arc_challenge/1022, arc_challenge/1024, arc_challenge/1025, arc_challenge/1026, arc_challenge/1028, arc_challenge/1030, arc_challenge/1031, arc_challenge/1034, arc_challenge/1038, arc_challenge/1042, arc_challenge/1045, arc_challenge/1049, arc_challenge/105, arc_challenge/1050, arc_challenge/1055, arc_challenge/1062, arc_challenge/1064, arc_challenge/1065, arc_challenge/1066, arc_challenge/1071, arc_challenge/1074, arc_challenge/1075, arc_challenge/1077, arc_challenge/1079, arc_challenge/1085, arc_challenge/1086, arc_challenge/1088, arc_challenge/109, arc_challenge/1095, arc_challenge/1096, arc_challenge/1097, arc_challenge/110, arc_challenge/1102, arc_challenge/1103, arc_challenge/1104, arc_challenge/1106, arc_challenge/1109, arc_challenge/111, arc_challenge/1110, arc_challenge/1111, arc_challenge/1114, arc_challenge/1115, arc_challenge/112, arc_challenge/1120, arc_challenge/1122, arc_challenge/1123, arc_challenge/1126, arc_challenge/113, arc_challenge/1130, arc_challenge/1132, arc_challenge/1133, arc_challenge/1135, arc_challenge/1137, arc_challenge/1138, arc_challenge/1140, arc_challenge/1143, arc_challenge/1146, arc_challenge/1147, arc_challenge/1148, arc_challenge/1152, arc_challenge/1157, arc_challenge/1160, arc_challenge/1161, arc_challenge/1164, arc_challenge/118, arc_challenge/119, arc_challenge/123, arc_challenge/125, arc_challenge/127, arc_challenge/129, arc_challenge/13, arc_challenge/130, arc_challenge/133, arc_challenge/136, arc_challenge/138, arc_challenge/14, arc_challenge/142, arc_challenge/144, arc_challenge/145, arc_challenge/149, arc_challenge/152, arc_challenge/154, arc_challenge/156, arc_challenge/158, arc_challenge/16, arc_challenge/162, arc_challenge/163, arc_challenge/167, arc_challenge/171, arc_challenge/172, arc_challenge/173, arc_challenge/174, arc_challenge/176, arc_challenge/177, arc_challenge/181, arc_challenge/182, arc_challenge/183, arc_challenge/185, arc_challenge/186, arc_challenge/189, arc_challenge/19, arc_challenge/191, arc_challenge/2, arc_challenge/20, arc_challenge/202, arc_challenge/204, arc_challenge/208, arc_challenge/209, arc_challenge/212, arc_challenge/218, arc_challenge/221, arc_challenge/222, arc_challenge/229, arc_challenge/230, arc_challenge/232, arc_challenge/233, arc_challenge/235, arc_challenge/236, arc_challenge/240, arc_challenge/241, arc_challenge/243, arc_challenge/247, arc_challenge/249, arc_challenge/250, arc_challenge/251, arc_challenge/252, arc_challenge/253, arc_challenge/254, arc_challenge/257, arc_challenge/259, arc_challenge/265, arc_challenge/266, arc_challenge/269, arc_challenge/27, arc_challenge/270, arc_challenge/271, arc_challenge/273, arc_challenge/276, arc_challenge/278, arc_challenge/28, arc_challenge/281, arc_challenge/284, arc_challenge/285, arc_challenge/286, arc_challenge/288, arc_challenge/289, arc_challenge/291, arc_challenge/293, arc_challenge/297, arc_challenge/298, arc_challenge/299, arc_challenge/303, arc_challenge/306, arc_challenge/307, arc_challenge/310, arc_challenge/311, arc_challenge/312, arc_challenge/313, arc_challenge/314, arc_challenge/315, arc_challenge/316, arc_challenge/317, arc_challenge/318, arc_challenge/324, arc_challenge/326, arc_challenge/327, arc_challenge/328, arc_challenge/329, arc_challenge/333, arc_challenge/337, arc_challenge/34, arc_challenge/345, arc_challenge/347, arc_challenge/35, arc_challenge/353, arc_challenge/357, arc_challenge/359, arc_challenge/361, arc_challenge/362, arc_challenge/363, arc_challenge/364, arc_challenge/366, arc_challenge/368, arc_challenge/369, arc_challenge/37, arc_challenge/370, arc_challenge/376, arc_challenge/379, arc_challenge/383, arc_challenge/389, arc_challenge/390, arc_challenge/391, arc_challenge/392, arc_challenge/394, arc_challenge/4, arc_challenge/400, arc_challenge/402, arc_challenge/404, arc_challenge/405, arc_challenge/406, arc_challenge/408, arc_challenge/41, arc_challenge/411, arc_challenge/413, arc_challenge/414, arc_challenge/417, arc_challenge/418, arc_challenge/419, arc_challenge/420, arc_challenge/423, arc_challenge/426, arc_challenge/427, arc_challenge/43, arc_challenge/432, arc_challenge/433, arc_challenge/435, arc_challenge/438, arc_challenge/440, arc_challenge/446, arc_challenge/447, arc_challenge/449, arc_challenge/45, arc_challenge/450, arc_challenge/456, arc_challenge/457, arc_challenge/459, arc_challenge/461, arc_challenge/462, arc_challenge/464, arc_challenge/466, arc_challenge/468, arc_challenge/47, arc_challenge/474, arc_challenge/475, arc_challenge/476, arc_challenge/477, arc_challenge/479, arc_challenge/48, arc_challenge/482, arc_challenge/483, arc_challenge/488, arc_challenge/489, arc_challenge/49, arc_challenge/492, arc_challenge/495, arc_challenge/497, arc_challenge/5, arc_challenge/50, arc_challenge/506, arc_challenge/508, arc_challenge/51, arc_challenge/513, arc_challenge/517, arc_challenge/518, arc_challenge/520, arc_challenge/522, arc_challenge/523, arc_challenge/524, arc_challenge/526, arc_challenge/53, arc_challenge/531, arc_challenge/532, arc_challenge/535, arc_challenge/54, arc_challenge/542, arc_challenge/545, arc_challenge/547, arc_challenge/548, arc_challenge/549, arc_challenge/55, arc_challenge/551, arc_challenge/554, arc_challenge/556, arc_challenge/56, arc_challenge/560, arc_challenge/569, arc_challenge/570, arc_challenge/574, arc_challenge/576, arc_challenge/577, arc_challenge/579, arc_challenge/580, arc_challenge/582, arc_challenge/585, arc_challenge/587, arc_challenge/588, arc_challenge/589, arc_challenge/592, arc_challenge/594, arc_challenge/597, arc_challenge/598, arc_challenge/599, arc_challenge/6, arc_challenge/602, arc_challenge/603, arc_challenge/604, arc_challenge/611, arc_challenge/613, arc_challenge/614, arc_challenge/620, arc_challenge/621, arc_challenge/622, arc_challenge/623, arc_challenge/624, arc_challenge/625, arc_challenge/626, arc_challenge/627, arc_challenge/629, arc_challenge/635, arc_challenge/64, arc_challenge/644, arc_challenge/649, arc_challenge/651, arc_challenge/652, arc_challenge/654, arc_challenge/655, arc_challenge/658, arc_challenge/660, arc_challenge/662, arc_challenge/664, arc_challenge/67, arc_challenge/670, arc_challenge/672, arc_challenge/674, arc_challenge/677, arc_challenge/678, arc_challenge/679, arc_challenge/682, arc_challenge/683, arc_challenge/684, arc_challenge/688, arc_challenge/690, arc_challenge/691, arc_challenge/692, arc_challenge/693, arc_challenge/694, arc_challenge/699, arc_challenge/70, arc_challenge/702, arc_challenge/704, arc_challenge/706, arc_challenge/708, arc_challenge/709, arc_challenge/71, arc_challenge/711, arc_challenge/715, arc_challenge/716, arc_challenge/717, arc_challenge/718, arc_challenge/72, arc_challenge/720, arc_challenge/723, arc_challenge/724, arc_challenge/726, arc_challenge/727, arc_challenge/728, arc_challenge/730, arc_challenge/732, arc_challenge/734, arc_challenge/738, arc_challenge/740, arc_challenge/743, arc_challenge/746, arc_challenge/747, arc_challenge/75, arc_challenge/751, arc_challenge/752, arc_challenge/753, arc_challenge/754, arc_challenge/755, arc_challenge/756, arc_challenge/757, arc_challenge/758, arc_challenge/759, arc_challenge/76, arc_challenge/765, arc_challenge/77, arc_challenge/774, arc_challenge/777, arc_challenge/78, arc_challenge/783, arc_challenge/789, arc_challenge/79, arc_challenge/790, arc_challenge/791, arc_challenge/792, arc_challenge/794, arc_challenge/796, arc_challenge/797, arc_challenge/798, arc_challenge/8, arc_challenge/800, arc_challenge/802, arc_challenge/803, arc_challenge/805, arc_challenge/808, arc_challenge/812, arc_challenge/813, arc_challenge/815, arc_challenge/816, arc_challenge/818, arc_challenge/820, arc_challenge/822, arc_challenge/826, arc_challenge/827, arc_challenge/828, arc_challenge/830, arc_challenge/835, arc_challenge/838, arc_challenge/84, arc_challenge/852, arc_challenge/855, arc_challenge/856, arc_challenge/857, arc_challenge/860, arc_challenge/861, arc_challenge/866, arc_challenge/873, arc_challenge/874, arc_challenge/875, arc_challenge/878, arc_challenge/879, arc_challenge/88, arc_challenge/880, arc_challenge/881, arc_challenge/882, arc_challenge/89, arc_challenge/890, arc_challenge/892, arc_challenge/893, arc_challenge/897, arc_challenge/901, arc_challenge/902, arc_challenge/903, arc_challenge/905, arc_challenge/908, arc_challenge/913, arc_challenge/914, arc_challenge/916, arc_challenge/918, arc_challenge/919, arc_challenge/921, arc_challenge/922, arc_challenge/923, arc_challenge/925, arc_challenge/926, arc_challenge/928, arc_challenge/93, arc_challenge/934, arc_challenge/943, arc_challenge/949, arc_challenge/953, arc_challenge/957, arc_challenge/959, arc_challenge/960, arc_challenge/961, arc_challenge/962, arc_challenge/965, arc_challenge/967, arc_challenge/971, arc_challenge/974, arc_challenge/977, arc_challenge/981, arc_challenge/982, arc_challenge/983, arc_challenge/987, arc_challenge/989, arc_challenge/990, arc_challenge/991, arc_challenge/996

Problems solved by 1 model only

example_link model min_pass1_of_model
arc_challenge/170 seed149 0.411
arc_challenge/160 seed149 0.411
arc_challenge/69 seed149 0.411
arc_challenge/850 seed149 0.411
arc_challenge/58 seed149 0.411
arc_challenge/424 seed149 0.411
arc_challenge/336 seed149 0.411
arc_challenge/279 seed149 0.411
arc_challenge/305 seed16385 0.404
arc_challenge/478 seed16385 0.404
arc_challenge/834 seed16385 0.404
arc_challenge/725 seed16385 0.404
arc_challenge/952 seed16385 0.404
arc_challenge/973 seed16385 0.404
arc_challenge/851 seed16385 0.404
arc_challenge/609 seed16385 0.404
arc_challenge/573 seed16385 0.404
arc_challenge/365 seed16385 0.404
arc_challenge/166 seed16385 0.404
arc_challenge/107 seed16385 0.404
arc_challenge/1056 seed24734 0.402
arc_challenge/537 seed24734 0.402
arc_challenge/242 seed24734 0.402
arc_challenge/843 seed24734 0.402
arc_challenge/999 seed24734 0.402
arc_challenge/1078 seed6529 0.399
arc_challenge/1 seed24762 0.399
arc_challenge/1083 seed24762 0.399
arc_challenge/917 seed24762 0.399
arc_challenge/92 seed24762 0.399
arc_challenge/912 seed6529 0.399
arc_challenge/776 seed6529 0.399
arc_challenge/615 seed24762 0.399
arc_challenge/737 seed6529 0.399
arc_challenge/760 seed24762 0.399
arc_challenge/1070 seed6529 0.399
arc_challenge/1129 seed24762 0.399
arc_challenge/1116 seed6529 0.399
arc_challenge/193 seed6529 0.399
arc_challenge/146 seed6529 0.399
arc_challenge/436 seed24762 0.399
arc_challenge/59 seed24762 0.399
arc_challenge/619 seed6529 0.399
arc_challenge/521 seed24762 0.399
arc_challenge/868 seed17495 0.396
arc_challenge/978 seed6115 0.396
arc_challenge/773 seed6115 0.396
arc_challenge/643 seed6115 0.396
arc_challenge/1047 seed17495 0.396
arc_challenge/102 seed17495 0.396
arc_challenge/1156 seed6115 0.396
arc_challenge/80 seed17495 0.396
arc_challenge/628 seed17495 0.396
arc_challenge/571 seed6115 0.396
arc_challenge/593 seed6115 0.396
arc_challenge/349 seed6115 0.396
arc_challenge/360 seed17495 0.396
arc_challenge/829 seed14309 0.394
arc_challenge/490 seed14309 0.394
arc_challenge/1058 seed30841 0.394
arc_challenge/1041 seed14309 0.394
arc_challenge/1029 seed14309 0.394
arc_challenge/1051 seed30841 0.394
arc_challenge/1001 seed30841 0.394
arc_challenge/992 seed30841 0.394
arc_challenge/215 seed30841 0.394
arc_challenge/322 seed14309 0.394
arc_challenge/343 seed30841 0.394
arc_challenge/39 seed30841 0.394
arc_challenge/352 seed30841 0.394
arc_challenge/401 seed30841 0.394
arc_challenge/595 seed14309 0.394
arc_challenge/994 seed30841 0.394
arc_challenge/942 seed14309 0.394
arc_challenge/964 seed30841 0.394
arc_challenge/584 seed30841 0.394
arc_challenge/632 seed14309 0.394
arc_challenge/65 seed30841 0.394
arc_challenge/729 seed30841 0.394
arc_challenge/795 seed30841 0.394
arc_challenge/872 seed30841 0.394
arc_challenge/159 seed27712 0.376
arc_challenge/1006 seed27712 0.376
arc_challenge/1068 seed27712 0.376
arc_challenge/501 seed27712 0.376
arc_challenge/425 seed27712 0.376
arc_challenge/685 seed27712 0.376
arc_challenge/939 seed27712 0.376

Suspect problems

These are 10 problems with the lowest correlation with the overall evaluation (i.e. better models tend to do worse on these. )

example_link pass1_of_ex tau
arc_challenge/66 0.200 -0.579
arc_challenge/681 0.200 -0.579
arc_challenge/1060 0.300 -0.572
arc_challenge/128 0.400 -0.472
arc_challenge/772 0.400 -0.472
arc_challenge/839 0.300 -0.471
arc_challenge/630 0.300 -0.471
arc_challenge/685 0.100 -0.463
arc_challenge/323 0.900 -0.463
arc_challenge/346 0.900 -0.463

Histogram of accuracies

Histogram of problems by the accuracy on each problem.

Histogram of difficulties

Histogram of problems by the minimum win rate to solve each problem.