swebench-verified: by models

Home Paper Code


SE predicted by accuracy

The typical standard errors between pairs of models on this dataset as a function of the absolute accuracy.

SE vs accuracy difference

The standard error of each model pair against their observed accuracy difference. Pairs below a reference line differ by more than the corresponding number of standard errors, i.e. they are statistically distinguishable at that level.

CDF of question level accuracy

Results table by model

model pass1 pass@count win_rate count SE(A) SE_x(A) SE_pred(A)
20250928_trae_doubao_seed_code 78.8 78.8 30.6 1 1.8 NaN NaN
20251120_livesweagent_gemini-3-pro-preview 77.4 77.4 29.3 1 1.9 NaN NaN
20250804_epam-ai-run-claude-4-sonnet 76.8 76.8 28.6 1 1.9 NaN NaN
20250902_atlassian-rovo-dev 76.8 76.8 28.5 1 1.9 NaN NaN
20250819_ACoder 76.4 76.4 28.2 1 1.9 NaN NaN
20250901_warp 75.6 75.6 28.1 1 1.9 NaN NaN
20250612_trae 75.2 75.2 27.2 1 1.9 NaN NaN
20251103_sonar-foundation-agent_claude-sonnet-4-5 74.8 74.8 27.4 1 1.9 NaN NaN
20250731_harness_ai 74.8 74.8 26.2 1 1.9 NaN NaN
20250720_Lingxi-v1.5_claude-4-sonnet-20250514 74.6 74.6 26.4 1 1.9 NaN NaN
20250915_JoyCode 74.6 74.6 27.3 1 1.9 NaN NaN
20251015_Prometheus_v1.2.1_gpt5 74.4 74.4 27.5 1 2 NaN NaN
20250603_Refact_Agent_claude-4-sonnet 74.4 74.4 26.4 1 2 NaN NaN
20251103_SalesforceAIResearch_SAGE_OpenHands 73.8 73.8 26.6 1 2 NaN NaN
20250522_tools_claude-4-opus 73.2 73.2 26.6 1 2 NaN NaN
20251021_SalesforceAIResearch_SAGE_bash_only 73 73 26.3 1 2 NaN NaN
20250522_tools_claude-4-sonnet 72.4 72.4 25.5 1 2 NaN NaN
20250807_openhands_gpt5 71.8 71.8 25.1 1 2 NaN NaN
20250715_qodo_command 71.2 71.2 24.5 1 2 NaN NaN
20250929_Prometheus_v1.2_gpt5 71.2 71.2 25.2 1 2 NaN NaN
20251014_Lingxi_kimi_k2 71.2 71.2 24.3 1 2 NaN NaN
20250710_bloop 71.2 71.2 24.3 1 2 NaN NaN
20250623_warp 71 71 24.4 1 2 NaN NaN
20250611_moatless_claude-4-sonnet-20250514 70.8 70.8 23.7 1 2 NaN NaN
20250519_trae 70.6 70.6 23.8 1 2 NaN NaN
20250610_augment_agent_v1 70.4 70.4 24.3 1 2 NaN NaN
20250515_Refact_Agent 70.4 70.4 23.5 1 2 NaN NaN
20250524_openhands_claude_4_sonnet 70.4 70.4 24.2 1 2 NaN NaN
20250519_devlo 70.2 70.2 23.4 1 2 NaN NaN
20250430_zencoder_ai 70 70 23.7 1 2 NaN NaN
20250805_openhands-Qwen3-Coder-480B-A35B-Instruct 69.6 69.6 23.7 1 2.1 NaN NaN
20250930_zai_glm4-6 68.2 68.2 22.6 1 2.1 NaN NaN
20250516_cortexa_o3 68.2 68.2 22.5 1 2.1 NaN NaN
20250522_sweagent_claude-4-sonnet-20250514 66.6 66.6 21.7 1 2.1 NaN NaN
20250514_aime_coder 66.4 66.4 21.3 1 2.1 NaN NaN
20250415_openhands 65.8 65.8 20.9 1 2.1 NaN NaN
20250316_augment_agent_v0 65.4 65.4 20.3 1 2.1 NaN NaN
20250716_openhands_kimi_k2 65.4 65.4 20.6 1 2.1 NaN NaN
20250405_amazon-q-developer-agent-20250405-dev 65.4 65.4 20.4 1 2.1 NaN NaN
20250503_patchpilot-v1.1-o4-mini 64.6 64.6 20.3 1 2.1 NaN NaN
20250117_wandb_programmer_o1_crosscheck5 64.6 64.6 20 1 2.1 NaN NaN
20250728_zai_glm4-5 64.2 64.2 20.1 1 2.1 NaN NaN
20250206_agentscope 63.4 63.4 18.8 1 2.2 NaN NaN
20250224_tools_claude-3-7-sonnet 63.2 63.2 19.4 1 2.2 NaN NaN
20250228_epam-ai-run-claude-3-5-sonnet 62.8 62.8 19.1 1 2.2 NaN NaN
20250110_blackboxai_agent_v1.1 62.8 62.8 19.8 1 2.2 NaN NaN
20250225_sweagent_claude-3-7-sonnet 62.4 62.4 18.6 1 2.2 NaN NaN
20241221_codestory_midwit_claude-3-5-sonnet_swe-search 62.2 62.2 18.6 1 2.2 NaN NaN
20250203_openhands_4x_scaled 60.8 60.8 17.7 1 2.2 NaN NaN
20250901_entroPO_R2E_QwenCoder30BA3B_tts 60.4 60.4 18.4 1 2.2 NaN NaN
20250110_learn_by_interact_claude3.5 60.2 60.2 20.2 1 2.2 NaN NaN
20250629_deepswerl_r2eagent_tts 58.8 58.8 17.3 1 2.2 NaN NaN
20241213_devlo 58.2 58.2 16.4 1 2.2 NaN NaN
20250410_cortexa 58.2 58.2 16.5 1 2.2 NaN NaN
20241223_emergent 57.2 57.2 15.5 1 2.2 NaN NaN
20241208_gru 57 57 15.8 1 2.2 NaN NaN
20250924_artemis_agent_v2 57 57 16.7 1 2.2 NaN NaN
20250405_swe-rizzo_claude37 56.6 56.6 15.9 1 2.2 NaN NaN
20241212_epam-ai-run-claude-3-5-sonnet 55.4 55.4 14.6 1 2.2 NaN NaN
20241202_amazon-q-developer-agent-20241202-dev 55 55 14.7 1 2.2 NaN NaN
20241108_devlo 54.2 54.2 14.4 1 2.2 NaN NaN
20251110_frogboss-32b 53.6 53.6 14.8 1 2.2 NaN NaN
20250804_codesweep_sweagent_kimi_k2_instruct 53.4 53.4 14.3 1 2.2 NaN NaN
20250120_Bracket 53.2 53.2 15.3 1 2.2 NaN NaN
20241029_OpenHands-CodeAct-2.1-sonnet-20241022 53 53 14.1 1 2.2 NaN NaN
20241212_google_jules_gemini_2.0_flash_experimental 52.2 52.2 14 1 2.2 NaN NaN
20250901_entroPO_R2E_QwenCoder30BA3B 52.2 52.2 13.9 1 2.2 NaN NaN
20241125_enginelabs 51.8 51.8 14.1 1 2.2 NaN NaN
20250805_openhands-Qwen3-Coder-30B-A3B-Instruct 51.6 51.6 13.4 1 2.2 NaN NaN
20250122_autocoderover-v2.1-claude-3-5-sonnet-20241022 51.6 51.6 13.4 1 2.2 NaN NaN
20241202_agentless-1.5_claude-3.5-sonnet-20241022 50.8 50.8 13.4 1 2.2 NaN NaN
20241125_marscode-agent-dev 50 50 12.9 1 2.2 NaN NaN
20241028_solver 50 50 12.5 1 2.2 NaN NaN
20241105_nfactorial 49.2 49.2 12.3 1 2.2 NaN NaN
20241022_tools_claude-3-5-sonnet-updated 49 49 12.3 1 2.2 NaN NaN
20241025_composio_swekit 48.6 48.6 11.9 1 2.2 NaN NaN
20241106_navie-2-gpt4o-sonnet 47.2 47.2 12.4 1 2.2 NaN NaN
20250616_Skywork-SWE-32B+TTS_Bo8 47 47 11.6 1 2.2 NaN NaN
20250520_openhands_devstral_small 46.8 46.8 11.5 1 2.2 NaN NaN
20241023_emergent 46.6 46.6 11.4 1 2.2 NaN NaN
20241108_autocoderover-v2.0-claude-3-5-sonnet-20241022 46.2 46.2 11.1 1 2.2 NaN NaN
20250528_patchpilot_Co-PatcheR 46 46 11 1 2.2 NaN NaN
20240924_solver 45.4 45.4 10.6 1 2.2 NaN NaN
20240824_gru 45.2 45.2 10.8 1 2.2 NaN NaN
20251110_frogmini-14b 45 45 10.8 1 2.2 NaN NaN
20250118_codeshellagent_gemini_2.0_flash_experimental 44.2 44.2 10.7 1 2.2 NaN NaN
20240920_solver 43.6 43.6 10.1 1 2.2 NaN NaN
20250214_agentless_lite_o3_mini 42.4 42.4 10.8 1 2.2 NaN NaN
20250527_amazon.nova-premier-v1.0 42.4 42.4 10.8 1 2.2 NaN NaN
20250629_deepswerl_r2eagent 42.2 42.2 10.7 1 2.2 NaN NaN
20250806_SWE-Exp_DeepSeek-V3 42 42 9.4 1 2.2 NaN NaN
20250112_ugaiforge 41.6 41.6 9.19 1 2.2 NaN NaN
20241030_nfactorial 41.6 41.6 9.89 1 2.2 NaN NaN
20250226_swerl_llama3_70b 41.2 41.2 9.81 1 2.2 NaN NaN
20240820_honeycomb 40.6 40.6 9.62 1 2.2 NaN NaN
20241016_composio_swekit 40.6 40.6 8.85 1 2.2 NaN NaN
20241022_tools_claude-3-5-haiku 40.6 40.6 9.1 1 2.2 NaN NaN
20241113_nebius-search-open-weight-models-11-24 40.6 40.6 8.92 1 2.2 NaN NaN
20250511_sweagent_lm_32b 40.2 40.2 8.71 1 2.2 NaN NaN
20241029_epam-ai-run-claude-3-5-sonnet 39.6 39.6 8.95 1 2.2 NaN NaN
20240721_amazon-q-developer-agent-20240719-dev 38.8 38.8 9.11 1 2.2 NaN NaN
20241028_agentless-1.5_gpt4o 38.8 38.8 8.72 1 2.2 NaN NaN
20240628_autocoderover-v20240620 38.4 38.4 8.98 1 2.2 NaN NaN
20250725_sweagent_devstral_small_2507 38 38 8.22 1 2.2 NaN NaN
20250616_Skywork-SWE-32B 38 38 8.51 1 2.2 NaN NaN
20240617_factory_code_droid 37 37 8.7 1 2.2 NaN NaN
20240620_sweagent_claude3.5sonnet 33.6 33.6 7.25 1 2.1 NaN NaN
20250306_SWE-Fixer_Qwen2.5-7b-retriever_Qwen2.5-72b-editor 32.8 32.8 6.95 1 2.1 NaN NaN
20240612_MASAI_gpt4o 32.6 32.6 6.99 1 2.1 NaN NaN
20241120_artemis_agent 32 32 6.74 1 2.1 NaN NaN
20241007_nfactorial 31.6 31.6 6.2 1 2.1 NaN NaN
20241128_SWE-Fixer_Qwen2.5-7b-retriever_Qwen2.5-72b-editor_20241128 30.2 30.2 6.2 1 2.1 NaN NaN
20241002_lingma-agent_lingma-swe-gpt-72b 28.8 28.8 5.9 1 2 NaN NaN
20241016_epam-ai-run-gpt-4o 27 27 5.54 1 2 NaN NaN
20240615_appmap-navie_gpt4o 26.2 26.2 5.17 1 2 NaN NaN
20241001_nfactorial 25.8 25.8 5.08 1 2 NaN NaN
20240509_amazon-q-developer-agent-20240430-dev 25.6 25.6 5.34 1 2 NaN NaN
20240918_lingma-agent_lingma-swe-gpt-72b 25 25 4.27 1 1.9 NaN NaN
20240820_epam-ai-run-gpt-4o 24 24 4.21 1 1.9 NaN NaN
20250627_agentless_MCTS-Refine-7B 23.2 23.2 6.11 1 1.9 NaN NaN
20240728_sweagent_gpt4o 23.2 23.2 4.17 1 1.9 NaN NaN
20240402_sweagent_gpt4 22.4 22.4 4 1 1.9 NaN NaN
20241002_lingma-agent_lingma-swe-gpt-7b 18.2 18.2 2.87 1 1.7 NaN NaN
20240402_sweagent_claude3opus 15.8 15.8 2.34 1 1.6 NaN NaN
20240918_lingma-agent_lingma-swe-gpt-7b 10.2 10.2 1.32 1 1.4 NaN NaN
20240402_rag_claude3opus 7 7 0.891 1 1.1 NaN NaN
20231010_rag_claude2 4.4 4.4 0.595 1 0.92 NaN NaN
20240402_rag_gpt4 2.8 2.8 0.355 1 0.74 NaN NaN
20231010_rag_swellama7b 1.4 1.4 0.398 1 0.53 NaN NaN
20231010_rag_swellama13b 1.2 1.2 0.252 1 0.49 NaN NaN
20231010_rag_gpt35 0.4 0.4 0.0585 1 0.28 NaN NaN