SWE-bench Pro
69 models
SWE-bench Pro — a benchmark by Scale AI (Xiang Deng, Jeff Da et al., arXiv 2509.16941). Contains 1,865 software engineering tasks from 41 actively maintained repositories (Python, Go, TypeScript, JavaScript). The agent must fix real bugs and implement features — multi-file patches with substantial code changes. Tasks take a professional engineer hours to days to complete. All tasks are human-verified. Score = % of issues resolved correctly.
Top 5 Models Performance
| Anthropic: claude-opus-5.5 | ######################################## | 89.9 |
| Anthropic: Claude Fable 5 | #################################### | 80.3 |
| Ornith: ornith-1.5-122b | ################################## | 76.9 |
| sakana: fugu-ultra | ################################# | 73.7 |
| Anthropic: Claude Opus 4.8 | ############################### | 69.2 |