humaneval_step27500: by examples

Results Paper Code


Not solved by any model

There are 138 examples not solved by any model. Solving some of these can be a good signal that your model is indeed better than leading models if these are good problems.
HumanEval/0, HumanEval/1, HumanEval/10, HumanEval/100, HumanEval/101, HumanEval/102, HumanEval/103, HumanEval/104, HumanEval/105, HumanEval/106, HumanEval/107, HumanEval/108, HumanEval/109, HumanEval/11, HumanEval/110, HumanEval/111, HumanEval/112, HumanEval/113, HumanEval/114, HumanEval/115, HumanEval/116, HumanEval/117, HumanEval/118, HumanEval/119, HumanEval/120, HumanEval/121, HumanEval/122, HumanEval/123, HumanEval/124, HumanEval/125, HumanEval/126, HumanEval/127, HumanEval/128, HumanEval/129, HumanEval/130, HumanEval/131, HumanEval/132, HumanEval/133, HumanEval/134, HumanEval/135, HumanEval/136, HumanEval/137, HumanEval/138, HumanEval/139, HumanEval/14, HumanEval/140, HumanEval/141, HumanEval/142, HumanEval/143, HumanEval/144, HumanEval/145, HumanEval/146, HumanEval/147, HumanEval/148, HumanEval/149, HumanEval/15, HumanEval/150, HumanEval/151, HumanEval/152, HumanEval/153, HumanEval/154, HumanEval/155, HumanEval/156, HumanEval/157, HumanEval/158, HumanEval/159, HumanEval/16, HumanEval/160, HumanEval/161, HumanEval/162, HumanEval/163, HumanEval/17, HumanEval/18, HumanEval/19, HumanEval/20, HumanEval/21, HumanEval/24, HumanEval/25, HumanEval/26, HumanEval/27, HumanEval/3, HumanEval/32, HumanEval/33, HumanEval/36, HumanEval/37, HumanEval/38, HumanEval/39, HumanEval/40, HumanEval/43, HumanEval/44, HumanEval/47, HumanEval/49, HumanEval/5, HumanEval/50, HumanEval/54, HumanEval/56, HumanEval/57, HumanEval/59, HumanEval/6, HumanEval/61, HumanEval/62, HumanEval/64, HumanEval/65, HumanEval/66, HumanEval/67, HumanEval/68, HumanEval/69, HumanEval/70, HumanEval/71, HumanEval/72, HumanEval/73, HumanEval/74, HumanEval/75, HumanEval/76, HumanEval/77, HumanEval/78, HumanEval/79, HumanEval/80, HumanEval/81, HumanEval/82, HumanEval/83, HumanEval/84, HumanEval/85, HumanEval/86, HumanEval/87, HumanEval/88, HumanEval/89, HumanEval/9, HumanEval/90, HumanEval/91, HumanEval/92, HumanEval/93, HumanEval/94, HumanEval/95, HumanEval/96, HumanEval/97, HumanEval/98, HumanEval/99

Problems solved by 1 model only

example_link model min_pass1_of_model
HumanEval/31 seed6529 0.110
HumanEval/51 seed6529 0.110
HumanEval/46 seed14309 0.098
HumanEval/63 seed14309 0.098
HumanEval/4 seed149 0.085
HumanEval/34 seed24762 0.079
HumanEval/41 seed24762 0.079

Suspect problems

These are 10 problems with the lowest correlation with the overall evaluation (i.e. better models tend to do worse on these. )

example_link pass1_of_ex tau
HumanEval/60 0.600 -0.351
HumanEval/12 0.400 -0.128
HumanEval/55 0.800 -0.117
HumanEval/41 0.100 -0.052
HumanEval/34 0.100 -0.052
HumanEval/2 0.200 -0.039
HumanEval/30 0.900 0.052
HumanEval/8 0.700 0.204
HumanEval/4 0.100 0.208
HumanEval/48 0.300 0.239

Histogram of accuracies

Histogram of problems by the accuracy on each problem.

Histogram of difficulties

Histogram of problems by the minimum win rate to solve each problem.