Evaluating a Code Agent: SWE-bench and Beyond
On this page
Lesson rewritten November 2, 2026. The previous version listed SWE-bench score tiers and a competitor table without sources; both have been removed. Devstral figures come from Mistral’s official announcement (December 9, 2025); SWE-bench formats come from the benchmark site. Devstral was removed from the API on July 31, 2026: its scores serve here as an example of how to read results. Sources: Devstral 2 and Mistral Vibe CLI (Mistral) · SWE-bench.
Measure an Agent, Not a Function
How do you objectively evaluate a model designed to solve development problems? Classic code generation benchmarks like HumanEval or MBPP verify whether a model can write an isolated function from a prompt. They don’t tell you if it can navigate a repository with hundreds of thousands of lines and fix a real bug. A model can excel at the first task and remain disappointing at the second.
SWE-bench fills this gap and has become the reference standard for code agents.
SWE-bench: Real Tickets, Real Tests
SWE-bench evaluates models on real GitHub tickets from popular Python repositories. The protocol replicates a developer’s workflow: the model receives a bug description or feature request without being told where to look; it explores the repository, locates the issue, and proposes a fix; that fix is then automatically judged by the project’s own tests—the same ones that protect the code in production.
The benchmark comes in several formats:
| Format | Content |
|---|---|
| SWE-bench (Full) | 2,294 tasks collected from 12 popular Python repositories |
| SWE-bench Lite | 300 tasks selected to reduce evaluation cost while preserving difficulty |
| SWE-bench Verified | 500 tasks filtered by humans, in collaboration with OpenAI |
The benchmark site also publishes Multilingual and Multimodal variants, useful if your code isn’t in Python or if your tickets include screenshots.
Reading a Score: The Devstral Example
A SWE-bench score of X% means the agent solved X% of the test set tickets—that is, it produced a fix that passes the project’s tests. It does not mean the model writes good code X% of the time.
At its launch on December 9, 2025, Mistral announced 72.2% on SWE-bench Verified for Devstral 2 (123 billion parameters) and 68.0% for Devstral Small 2 (24 billion). The gap is small relative to the size difference: this is what made Devstral Small 2 interesting, and why Vibe’s documentation still recommends it today for local use.
The same announcement illustrates an essential caveat. Mistral claimed up to seven times lower cost than Claude Sonnet on real tasks, while noting that human evaluators still strongly preferred Claude Sonnet 4.5 over Devstral 2. A benchmark score, a cost figure, and human preference measure three different things; none replaces the other two.
What SWE-bench Doesn’t Tell You
SWE-bench tests a specific style of work: fixing bugs in open source Python projects with an existing test suite. Your daily work in TypeScript, Java, or on an application without tests may have nothing to do with this setup.
Evaluation conditions carry heavy weight: the number of allowed attempts, the tooling (scaffolding) provided to the agent, and time limits can shift scores by several points. Two figures published separately by two vendors with two different tooling setups are therefore not always comparable. Prefer rankings that evaluate models in a common framework, and always read the methodology before the number.
Size matters too, and not just for the bill: a 24-billion-parameter model that fits on a single GPU is usable where a much larger model requires a cluster and a team to maintain.
Your Own Benchmark
The best evaluation is the one you build on your own repository. Take three to five already-resolved tickets, along with the fix your team wrote and the associated tests. Have candidate models process them—for example, Mistral Small 4 and Large 4 via the API, or Vibe Code with its built-in model—then compare:
- Success rate: Do the tests pass with the proposed fix?
- Diff quality: Is it minimal, readable, and compliant with your conventions?
- Cost and time: Tokens consumed, session duration, number of human interventions.
No public ranking will replace this test, and you can rerun it in an hour with every new model.
Key Takeaways
- SWE-bench evaluates agents on real GitHub tickets, judged by project tests
- Three main formats: Full (2,294 tasks), Lite (300), Verified (500, human-filtered)
- Devstral 2 and Devstral Small 2 announced 72.2% and 68.0% on SWE-bench Verified
- Score, cost, and human preference measure different things; evaluation tooling changes the numbers
- The decisive test runs on your own resolved tickets