gsm8k_step02500: by models

Home Paper Code


SE predicted by accuracy

The typical standard errors between pairs of models on this dataset as a function of the absolute accuracy.

SE vs accuracy difference

The standard error of each model pair against their observed accuracy difference. Pairs below a reference line differ by more than the corresponding number of standard errors, i.e. they are statistically distinguishable at that level.

CDF of question level accuracy

Results table by model

model pass1 pass@count win_rate count SE(A) SE_x(A) SE_pred(A)
seed30841 1.9 1.9 1.51 1 0.38 NaN NaN
seed16385 1.74 1.74 1.31 1 0.36 NaN NaN
seed6529 1.67 1.67 1.14 1 0.35 NaN NaN
seed17495 1.59 1.59 1.18 1 0.34 NaN NaN
seed24734 1.59 1.59 1.29 1 0.34 NaN NaN
seed6115 1.52 1.52 1.04 1 0.34 NaN NaN
seed149 1.36 1.36 0.935 1 0.32 NaN NaN
seed24762 1.29 1.29 0.927 1 0.31 NaN NaN
seed14309 1.14 1.14 0.733 1 0.29 NaN NaN
seed27712 1.14 1.14 0.775 1 0.29 NaN NaN