There are 132 examples not solved by any model.
Solving some of these can be a good signal that your model is indeed better than leading models if these are good problems.
HumanEval/0, HumanEval/1, HumanEval/10, HumanEval/100, HumanEval/101, HumanEval/102, HumanEval/103, HumanEval/104, HumanEval/105, HumanEval/106, HumanEval/107, HumanEval/108, HumanEval/109, HumanEval/11, HumanEval/110, HumanEval/111, HumanEval/112, HumanEval/113, HumanEval/114, HumanEval/115, HumanEval/116, HumanEval/117, HumanEval/118, HumanEval/119, HumanEval/120, HumanEval/121, HumanEval/122, HumanEval/123, HumanEval/124, HumanEval/125, HumanEval/126, HumanEval/127, HumanEval/128, HumanEval/129, HumanEval/130, HumanEval/131, HumanEval/132, HumanEval/133, HumanEval/134, HumanEval/135, HumanEval/136, HumanEval/137, HumanEval/138, HumanEval/139, HumanEval/14, HumanEval/140, HumanEval/141, HumanEval/142, HumanEval/143, HumanEval/144, HumanEval/145, HumanEval/146, HumanEval/147, HumanEval/148, HumanEval/149, HumanEval/150, HumanEval/151, HumanEval/153, HumanEval/154, HumanEval/155, HumanEval/156, HumanEval/157, HumanEval/158, HumanEval/159, HumanEval/16, HumanEval/160, HumanEval/161, HumanEval/162, HumanEval/163, HumanEval/17, HumanEval/18, HumanEval/19, HumanEval/20, HumanEval/24, HumanEval/25, HumanEval/26, HumanEval/3, HumanEval/32, HumanEval/33, HumanEval/36, HumanEval/37, HumanEval/38, HumanEval/39, HumanEval/4, HumanEval/40, HumanEval/41, HumanEval/43, HumanEval/44, HumanEval/47, HumanEval/5, HumanEval/50, HumanEval/56, HumanEval/57, HumanEval/59, HumanEval/6, HumanEval/61, HumanEval/62, HumanEval/64, HumanEval/65, HumanEval/66, HumanEval/67, HumanEval/68, HumanEval/69, HumanEval/70, HumanEval/71, HumanEval/72, HumanEval/73, HumanEval/74, HumanEval/75, HumanEval/76, HumanEval/77, HumanEval/78, HumanEval/79, HumanEval/81, HumanEval/82, HumanEval/83, HumanEval/84, HumanEval/85, HumanEval/86, HumanEval/87, HumanEval/88, HumanEval/89, HumanEval/90, HumanEval/91, HumanEval/92, HumanEval/93, HumanEval/94, HumanEval/95, HumanEval/96, HumanEval/97, HumanEval/98, HumanEval/99
| example_link | model | min_pass1_of_model |
|---|---|---|
| HumanEval/13 | seed27712 | 0.140 |
| HumanEval/51 | seed6115 | 0.134 |
| HumanEval/54 | seed6115 | 0.134 |
| HumanEval/9 | seed24734 | 0.128 |
| HumanEval/21 | seed17495 | 0.116 |
| HumanEval/80 | seed30841 | 0.091 |
These are 10 problems with the lowest correlation with the overall evaluation (i.e. better models tend to do worse on these. )
| example_link | pass1_of_ex | tau |
|---|---|---|
| HumanEval/80 | 0.100 | -0.416 |
| HumanEval/60 | 0.900 | -0.416 |
| HumanEval/21 | 0.100 | -0.104 |
| HumanEval/8 | 0.900 | -0.104 |
| HumanEval/15 | 0.200 | 0.000 |
| HumanEval/152 | 0.400 | 0.096 |
| HumanEval/9 | 0.100 | 0.104 |
| HumanEval/45 | 0.900 | 0.104 |
| HumanEval/46 | 0.500 | 0.156 |
| HumanEval/49 | 0.500 | 0.156 |
Histogram of problems by the accuracy on each problem.
Histogram of problems by the minimum win rate to solve each problem.