SWE-bench is a benchmark of real issues from open-source Python repositories: the model, driven by an agent harness, gets the repository and the issue text and must produce a patch that makes the issue's hidden tests pass without breaking others. SWE-bench Verified is a subset of 500 tasks checked by humans to be solvable and fairly tested; its score is the share resolved.
At a glance
Task
Real GitHub issue in, patch out, judged by the project's tests
Verified
500 human-checked tasks, the usual headline number
Measures
Model plus harness, not the model alone
Blind spot
Python only, and scores depend heavily on the agent around the model
Why do two SWE-bench numbers for the same model disagree?
Because the benchmark scores a model inside an agent: the harness, its prompts,
the number of attempts and the tools it offers all move the score. A vendor’s own
harness, a leaderboard’s standard harness and your coding CLI will give different
results for the same weights. Compare numbers only when the setup is the same,
and treat a vendor’s self-reported score as a claim until someone reproduces it.
How should I use it when choosing a local model?
As a filter, not a verdict. It narrows the candidates to models that can work on
real code at all; then a small benchmark on your own tasks, in your own harness,
on your own hardware decides. That second step regularly reorders the list.
SWE-bench tells you
How often a setup turns an issue into a passing patch