mbpp+: by models

Home Paper Code


SE predicted by accuracy

The typical standard errors between pairs of models on this dataset as a function of the absolute accuracy.

SE vs accuracy difference

The standard error of each model pair against their observed accuracy difference. Pairs below a reference line differ by more than the corresponding number of standard errors, i.e. they are statistically distinguishable at that level.

CDF of question level accuracy

Results table by model

model pass1 pass@count win_rate count SE(A) SE_x(A) SE_pred(A)
gpt-4-1106-preview 74.1 74.1 23.9 1 2.3 NaN NaN
claude-3-opus-20240229 73.5 73.5 23.1 1 2.3 NaN NaN
deepseek-coder-33b-instruct 70.4 70.4 21.2 1 2.3 NaN NaN
claude-3-sonnet-20240229 69.8 69.8 20.2 1 2.4 NaN NaN
meta-llama-3-70b-instruct 69.6 69.6 19.3 1 2.4 NaN NaN
claude-3-haiku-20240307 69.3 69.3 20.1 1 2.4 NaN NaN
opencodeinterpreter-ds-33b 68.8 68.8 19.2 1 2.4 NaN NaN
white-rabbit-neo-33b-v1 67.5 67.5 18.5 1 2.4 NaN NaN
opencodeinterpreter-ds-6.7b 66.9 66.9 17.6 1 2.4 NaN NaN
xwincoder-34b 66.1 66.1 16.9 1 2.4 NaN NaN
deepseek-coder-6.7b-instruct 66.1 66.1 17.6 1 2.4 NaN NaN
bigcode--starcoder2-15b-instruct-v0.1 65.1 65.1 17.6 1 2.5 NaN NaN
HuggingFaceH4--starchat2-15b-v0.1 64.8 64.8 16.9 1 2.5 NaN NaN
code-millenials-34b 64.6 64.6 16.7 1 2.5 NaN NaN
mixtral-8x22b-instruct-v0.1 64.6 64.6 16.9 1 2.5 NaN NaN
wizardcoder-34b 63.8 63.8 15.9 1 2.5 NaN NaN
CohereForAI--c4ai-command-r-plus 63.8 63.8 17.1 1 2.5 NaN NaN
starcoder2-15b-oci 63.8 63.8 16.2 1 2.5 NaN NaN
speechless-starcoder2-15b 63 63 15.2 1 2.5 NaN NaN
Qwen--Qwen1.5-72B-Chat 62.4 62.4 15.4 1 2.5 NaN NaN
speechless-codellama-34b 61.4 61.4 14.5 1 2.5 NaN NaN
dolphin-2.6 60.1 60.1 13.9 1 2.5 NaN NaN
mistral-large-latest 59.8 59.8 15.8 1 2.5 NaN NaN
deepseek-coder-6.7b-base 59.5 59.5 13.7 1 2.5 NaN NaN
codegemma-7b-it 57.4 57.4 12.9 1 2.5 NaN NaN
speechless-starcoder2-7b 57.1 57.1 12.6 1 2.5 NaN NaN
code-llama-34b 56.9 56.9 12.9 1 2.5 NaN NaN
databricks--dbrx-instruct 56.3 56.3 13.8 1 2.6 NaN NaN
openchat 56.1 56.1 12.2 1 2.6 NaN NaN
phi-2 55.3 55.3 12 1 2.6 NaN NaN
code-llama-multi-34b 55 55 11.6 1 2.6 NaN NaN
wizardcoder-15b 54.8 54.8 11.6 1 2.6 NaN NaN
microsoft--Phi-3-mini-4k-instruct 54.5 54.5 13.5 1 2.6 NaN NaN
code-llama-multi-13b 54.5 54.5 11.4 1 2.6 NaN NaN
code-llama-13b 53.2 53.2 11.1 1 2.6 NaN NaN
codegemma-7b 52.4 52.4 11.4 1 2.6 NaN NaN
octocoder 51.3 51.3 10 1 2.6 NaN NaN
mixtral-8x7b-instruct 50.3 50.3 11.9 1 2.6 NaN NaN
wizardcoder-7b 50 50 9.48 1 2.6 NaN NaN
speechless-mistral-7b 49.2 49.2 10.2 1 2.6 NaN NaN
codet5p-16b 48.1 48.1 8.65 1 2.6 NaN NaN
codegemma-2b 47.9 47.9 9.53 1 2.6 NaN NaN
stable-code-3b 46.8 46.8 8.47 1 2.6 NaN NaN
open-hermes-2.5-code-290k-13b 46.8 46.8 10.1 1 2.6 NaN NaN
gemma-1.1-7b-it 46.6 46.6 8.85 1 2.6 NaN NaN
codegen-16b 46.3 46.3 8.51 1 2.6 NaN NaN
gemma-7b 45 45 8.73 1 2.6 NaN NaN
starcoder2-3b 44.4 44.4 8.26 1 2.6 NaN NaN
code-llama-multi-7b 44.2 44.2 7.61 1 2.6 NaN NaN
codegen-6b 43.7 43.7 7.7 1 2.6 NaN NaN
mistral-7b 42.9 42.9 7.01 1 2.5 NaN NaN
codet5p-6b 42.6 42.6 7.82 1 2.5 NaN NaN
xdan-l1-chat 41.8 41.8 7.6 1 2.5 NaN NaN
codet5p-2b 38.9 38.9 6.16 1 2.5 NaN NaN
mistralai--Mistral-7B-Instruct-v0.2 37.6 37.6 7 1 2.5 NaN NaN
solar-10.7b-instruct 37.6 37.6 6.41 1 2.5 NaN NaN
codegen-2b 37.3 37.3 5.65 1 2.5 NaN NaN
gemma-2b 35.4 35.4 5.42 1 2.5 NaN NaN
gemma-7b-it 33.1 33.1 6.01 1 2.4 NaN NaN