There are 159 examples not solved by any model.
Solving some of these can be a good signal that your model is indeed better than leading models if these are good problems.
HumanEval/0, HumanEval/1, HumanEval/10, HumanEval/100, HumanEval/101, HumanEval/102, HumanEval/103, HumanEval/104, HumanEval/105, HumanEval/106, HumanEval/107, HumanEval/108, HumanEval/109, HumanEval/11, HumanEval/110, HumanEval/111, HumanEval/112, HumanEval/113, HumanEval/114, HumanEval/115, HumanEval/116, HumanEval/117, HumanEval/118, HumanEval/119, HumanEval/12, HumanEval/120, HumanEval/121, HumanEval/122, HumanEval/123, HumanEval/124, HumanEval/125, HumanEval/126, HumanEval/127, HumanEval/128, HumanEval/129, HumanEval/13, HumanEval/130, HumanEval/131, HumanEval/132, HumanEval/133, HumanEval/134, HumanEval/135, HumanEval/136, HumanEval/137, HumanEval/138, HumanEval/139, HumanEval/14, HumanEval/140, HumanEval/141, HumanEval/142, HumanEval/143, HumanEval/144, HumanEval/145, HumanEval/146, HumanEval/147, HumanEval/148, HumanEval/149, HumanEval/15, HumanEval/150, HumanEval/151, HumanEval/152, HumanEval/153, HumanEval/154, HumanEval/155, HumanEval/156, HumanEval/157, HumanEval/158, HumanEval/159, HumanEval/16, HumanEval/160, HumanEval/161, HumanEval/162, HumanEval/163, HumanEval/17, HumanEval/18, HumanEval/19, HumanEval/2, HumanEval/20, HumanEval/21, HumanEval/22, HumanEval/24, HumanEval/25, HumanEval/26, HumanEval/27, HumanEval/29, HumanEval/3, HumanEval/30, HumanEval/31, HumanEval/32, HumanEval/33, HumanEval/36, HumanEval/37, HumanEval/38, HumanEval/39, HumanEval/4, HumanEval/40, HumanEval/41, HumanEval/42, HumanEval/43, HumanEval/44, HumanEval/45, HumanEval/46, HumanEval/47, HumanEval/48, HumanEval/49, HumanEval/5, HumanEval/50, HumanEval/51, HumanEval/52, HumanEval/54, HumanEval/55, HumanEval/56, HumanEval/57, HumanEval/58, HumanEval/59, HumanEval/6, HumanEval/60, HumanEval/61, HumanEval/62, HumanEval/63, HumanEval/64, HumanEval/65, HumanEval/66, HumanEval/67, HumanEval/68, HumanEval/69, HumanEval/7, HumanEval/70, HumanEval/71, HumanEval/72, HumanEval/73, HumanEval/74, HumanEval/75, HumanEval/76, HumanEval/77, HumanEval/78, HumanEval/79, HumanEval/8, HumanEval/80, HumanEval/81, HumanEval/82, HumanEval/83, HumanEval/84, HumanEval/85, HumanEval/86, HumanEval/87, HumanEval/88, HumanEval/89, HumanEval/9, HumanEval/90, HumanEval/91, HumanEval/92, HumanEval/93, HumanEval/94, HumanEval/95, HumanEval/96, HumanEval/97, HumanEval/98, HumanEval/99
| example_link | model | min_pass1_of_model |
|---|---|---|
| HumanEval/34 | seed24734 | 0.012 |
| HumanEval/23 | seed14309 | 0.006 |
| HumanEval/28 | seed149 | 0.006 |
These are 10 problems with the lowest correlation with the overall evaluation (i.e. better models tend to do worse on these. )
| example_link | pass1_of_ex | tau |
|---|---|---|
| HumanEval/23 | 0.100 | 0.070 |
| HumanEval/28 | 0.100 | 0.070 |
| HumanEval/35 | 0.200 | 0.104 |
| HumanEval/53 | 0.400 | 0.511 |
| HumanEval/34 | 0.100 | 0.626 |
| HumanEval/0 | 0.000 | NaN |
| HumanEval/1 | 0.000 | NaN |
| HumanEval/10 | 0.000 | NaN |
| HumanEval/100 | 0.000 | NaN |
| HumanEval/101 | 0.000 | NaN |
Histogram of problems by the accuracy on each problem.
Histogram of problems by the minimum win rate to solve each problem.