mbpp: by models

Home Paper Code


SE predicted by accuracy

The typical standard errors between pairs of models on this dataset as a function of the absolute accuracy.

SE vs accuracy difference

The standard error of each model pair against their observed accuracy difference. Pairs below a reference line differ by more than the corresponding number of standard errors, i.e. they are statistically distinguishable at that level.

CDF of question level accuracy

Results table by model

model pass1 pass@count win_rate count SE(A) SE_x(A) SE_pred(A)
claude-3-opus-20240229 89.4 89.4 26.5 1 1.6 NaN NaN
gpt-4-1106-preview 85.7 85.7 24.4 1 1.8 NaN NaN
claude-3-sonnet-20240229 83.6 83.6 22.5 1 1.9 NaN NaN
meta-llama-3-70b-instruct 82.3 82.3 21.3 1 2 NaN NaN
deepseek-coder-33b-instruct 80.4 80.4 21.3 1 2 NaN NaN
claude-3-haiku-20240307 80.2 80.2 21.3 1 2.1 NaN NaN
opencodeinterpreter-ds-33b 80.2 80.2 20.6 1 2.1 NaN NaN
white-rabbit-neo-33b-v1 79.4 79.4 19.8 1 2.1 NaN NaN
bigcode--starcoder2-15b-instruct-v0.1 78 78 19.1 1 2.1 NaN NaN
xwincoder-34b 77 77 18.1 1 2.2 NaN NaN
opencodeinterpreter-ds-6.7b 76.5 76.5 17.7 1 2.2 NaN NaN
code-millenials-34b 76.2 76.2 17.5 1 2.2 NaN NaN
wizardcoder-34b 75.1 75.1 17.5 1 2.2 NaN NaN
deepseek-coder-6.7b-instruct 74.9 74.9 17 1 2.2 NaN NaN
HuggingFaceH4--starchat2-15b-v0.1 74.9 74.9 17.4 1 2.2 NaN NaN
starcoder2-15b-oci 74.3 74.3 16.8 1 2.2 NaN NaN
CohereForAI--c4ai-command-r-plus 74.3 74.3 17.7 1 2.2 NaN NaN
mixtral-8x22b-instruct-v0.1 73.8 73.8 17.1 1 2.3 NaN NaN
speechless-codellama-34b 73.8 73.8 16.6 1 2.3 NaN NaN
speechless-starcoder2-15b 73.5 73.5 16.1 1 2.3 NaN NaN
mistral-large-latest 72.8 72.8 17.9 1 2.3 NaN NaN
Qwen--Qwen1.5-72B-Chat 72.5 72.5 15.5 1 2.3 NaN NaN
deepseek-coder-6.7b-base 72 72 15.5 1 2.3 NaN NaN
dolphin-2.6 70.6 70.6 14.9 1 2.3 NaN NaN
codegemma-7b-it 70.4 70.4 15.1 1 2.3 NaN NaN
code-llama-34b 69.3 69.3 14.6 1 2.4 NaN NaN
databricks--dbrx-instruct 67.2 67.2 14.3 1 2.4 NaN NaN
speechless-starcoder2-7b 66.7 66.7 13.6 1 2.4 NaN NaN
code-llama-multi-34b 66.7 66.7 13.2 1 2.4 NaN NaN
microsoft--Phi-3-mini-4k-instruct 65.9 65.9 14.7 1 2.4 NaN NaN
codegemma-7b 65.1 65.1 13.2 1 2.5 NaN NaN
wizardcoder-15b 64.3 64.3 12 1 2.5 NaN NaN
phi-2 64 64 12.4 1 2.5 NaN NaN
openchat 63.8 63.8 12 1 2.5 NaN NaN
code-llama-13b 63.5 63.5 12 1 2.5 NaN NaN
code-llama-multi-13b 63 63 11.5 1 2.5 NaN NaN
mixtral-8x7b-instruct 59.5 59.5 12.3 1 2.5 NaN NaN
octocoder 59.3 59.3 9.86 1 2.5 NaN NaN
wizardcoder-7b 58.5 58.5 10 1 2.5 NaN NaN
speechless-mistral-7b 57.4 57.4 10.3 1 2.5 NaN NaN
gemma-1.1-7b-it 57.1 57.1 10.6 1 2.5 NaN NaN
codet5p-16b 56.6 56.6 8.68 1 2.5 NaN NaN
codegemma-2b 55.6 55.6 9 1 2.6 NaN NaN
stable-code-3b 54.8 54.8 8.35 1 2.6 NaN NaN
codegen-16b 54.2 54.2 8.82 1 2.6 NaN NaN
code-llama-multi-7b 53.7 53.7 8.69 1 2.6 NaN NaN
starcoder2-3b 53.4 53.4 9.68 1 2.6 NaN NaN
codet5p-6b 52.9 52.9 8.88 1 2.6 NaN NaN
gemma-7b 52.6 52.6 8.61 1 2.6 NaN NaN
open-hermes-2.5-code-290k-13b 52.4 52.4 9.1 1 2.6 NaN NaN
mistral-7b 51.9 51.9 7.37 1 2.6 NaN NaN
codegen-6b 50.8 50.8 7.68 1 2.6 NaN NaN
xdan-l1-chat 50.3 50.3 8.23 1 2.6 NaN NaN
codet5p-2b 48.4 48.4 7.5 1 2.6 NaN NaN
codegen-2b 46.3 46.3 6.97 1 2.6 NaN NaN
mistralai--Mistral-7B-Instruct-v0.2 44.7 44.7 6.68 1 2.6 NaN NaN
solar-10.7b-instruct 43.9 43.9 6.21 1 2.6 NaN NaN
gemma-2b 41.8 41.8 5.29 1 2.5 NaN NaN
gemma-7b-it 39.7 39.7 6.51 1 2.5 NaN NaN