humaneval_step50000: by examples

Results Paper Code


Not solved by any model

There are 132 examples not solved by any model. Solving some of these can be a good signal that your model is indeed better than leading models if these are good problems.
HumanEval/0, HumanEval/1, HumanEval/10, HumanEval/100, HumanEval/101, HumanEval/102, HumanEval/103, HumanEval/104, HumanEval/105, HumanEval/106, HumanEval/107, HumanEval/108, HumanEval/109, HumanEval/11, HumanEval/110, HumanEval/111, HumanEval/112, HumanEval/113, HumanEval/114, HumanEval/115, HumanEval/116, HumanEval/117, HumanEval/118, HumanEval/119, HumanEval/120, HumanEval/121, HumanEval/122, HumanEval/123, HumanEval/124, HumanEval/125, HumanEval/126, HumanEval/127, HumanEval/128, HumanEval/129, HumanEval/130, HumanEval/131, HumanEval/132, HumanEval/133, HumanEval/134, HumanEval/135, HumanEval/136, HumanEval/137, HumanEval/138, HumanEval/139, HumanEval/14, HumanEval/140, HumanEval/141, HumanEval/142, HumanEval/143, HumanEval/144, HumanEval/145, HumanEval/146, HumanEval/147, HumanEval/148, HumanEval/149, HumanEval/150, HumanEval/151, HumanEval/153, HumanEval/154, HumanEval/155, HumanEval/156, HumanEval/157, HumanEval/158, HumanEval/159, HumanEval/16, HumanEval/160, HumanEval/161, HumanEval/162, HumanEval/163, HumanEval/17, HumanEval/18, HumanEval/19, HumanEval/20, HumanEval/24, HumanEval/25, HumanEval/26, HumanEval/3, HumanEval/32, HumanEval/33, HumanEval/36, HumanEval/37, HumanEval/38, HumanEval/39, HumanEval/4, HumanEval/40, HumanEval/41, HumanEval/43, HumanEval/44, HumanEval/47, HumanEval/5, HumanEval/50, HumanEval/56, HumanEval/57, HumanEval/59, HumanEval/6, HumanEval/61, HumanEval/62, HumanEval/64, HumanEval/65, HumanEval/66, HumanEval/67, HumanEval/68, HumanEval/69, HumanEval/70, HumanEval/71, HumanEval/72, HumanEval/73, HumanEval/74, HumanEval/75, HumanEval/76, HumanEval/77, HumanEval/78, HumanEval/79, HumanEval/81, HumanEval/82, HumanEval/83, HumanEval/84, HumanEval/85, HumanEval/86, HumanEval/87, HumanEval/88, HumanEval/89, HumanEval/90, HumanEval/91, HumanEval/92, HumanEval/93, HumanEval/94, HumanEval/95, HumanEval/96, HumanEval/97, HumanEval/98, HumanEval/99

Problems solved by 1 model only

example_link model min_pass1_of_model
HumanEval/13 seed27712 0.140
HumanEval/51 seed6115 0.134
HumanEval/54 seed6115 0.134
HumanEval/9 seed24734 0.128
HumanEval/21 seed17495 0.116
HumanEval/80 seed30841 0.091

Suspect problems

These are 10 problems with the lowest correlation with the overall evaluation (i.e. better models tend to do worse on these. )

example_link pass1_of_ex tau
HumanEval/80 0.100 -0.416
HumanEval/60 0.900 -0.416
HumanEval/21 0.100 -0.104
HumanEval/8 0.900 -0.104
HumanEval/15 0.200 0.000
HumanEval/152 0.400 0.096
HumanEval/9 0.100 0.104
HumanEval/45 0.900 0.104
HumanEval/46 0.500 0.156
HumanEval/49 0.500 0.156

Histogram of accuracies

Histogram of problems by the accuracy on each problem.

Histogram of difficulties

Histogram of problems by the minimum win rate to solve each problem.