Measuring all the noises of LLM evals

Home Paper Code

Datasets: main, train_curve, training_seeds, Open models with vLLM: T=0.7, T=1, high K T=0.7,

Details are linked and more summary stats are available on mouse over.

Full table, Descriptions

benchmark_id size models SE(A) SE(A-B) no_solve tau- details
agi_english_step02500 2546 10 0.81 0.96 35 30 models | examples | data | more
agi_english_step27500 2546 10 0.83 1.1 26 36 models | examples | data | more
agi_english_step50000 2546 10 0.84 1.1 26 33 models | examples | data | more
arc_challenge_step02500 1165 10 1.3 0.91 61 10 models | examples | data | more
arc_challenge_step27500 1165 10 1.4 1.1 44 14 models | examples | data | more
arc_challenge_step50000 1165 10 1.4 1.1 41 16 models | examples | data | more
gsm8k_step02500 1319 10 0.33 0.4 93 2.5 models | examples | data | more
gsm8k_step27500 1319 10 0.45 0.6 82 6 models | examples | data | more
gsm8k_step50000 1319 10 0.54 0.68 79 7.9 models | examples | data | more
hellaswag_step02500 10042 10 0.48 0.3 50 11 models | examples | data | more
hellaswag_step27500 10042 10 0.47 0.28 24 9.6 models | examples | data | more
hellaswag_step50000 10042 10 0.46 0.26 21 8.8 models | examples | data | more
humaneval_step02500 164 10 0.64 0.74 97 0 models | examples | data | more
humaneval_step27500 164 10 2.1 1.6 84 3.7 models | examples | data | more
humaneval_step50000 164 10 2.5 1.8 80 2.4 models | examples | data | more
nq_step02500 3610 10 0.22 0.23 94 1.8 models | examples | data | more
nq_step27500 3610 10 0.56 0.51 71 10 models | examples | data | more
nq_step50000 3610 10 0.62 0.54 67 11 models | examples | data | more
piqa_step02500 1838 10 1.1 0.76 22 13 models | examples | data | more
piqa_step27500 1838 10 1 0.68 15 9.2 models | examples | data | more
piqa_step50000 1838 10 0.98 0.65 15 9 models | examples | data | more
siqa_step02500 1954 10 1.1 0.66 46 9.6 models | examples | data | more
siqa_step27500 1954 10 1.1 0.68 42 9.5 models | examples | data | more
siqa_step50000 1954 10 1.1 0.67 42 10 models | examples | data | more
tqa_step02500 11313 10 0.19 0.19 88 4 models | examples | data | more
tqa_step27500 11313 10 0.45 0.35 46 16 models | examples | data | more
tqa_step50000 11313 10 0.47 0.35 38 16 models | examples | data | more

Notes:


Generated on 2026-10-13 22:06:46