Our Models Are (Sometimes) Better Than Our Benchmarks
I mentioned previously that I spent a while updating my own harness code review process, and that I had found disappointing results from SWR-Bench. Since I have now published my resulting fork of SWRBench, I think it’s time to talk a little more about what that experience was, how I followed up on it, and why I think it’s important.
Finding Benchmark Flaws
First, my experience. I was chatting with Claude CLI about my code review process. I searched for code review benchmarks, and found SWR-Bench. I cloned the repo to try running my code review process against the benchmark. I isolated a single PR to have my system review to ensure the piping was correct before committing to more spending and delay.
The first PR in alphabetical order was astropy-1010, a PR trying to address a bug in unicode/string parsing. The PR adds an attempt to parse the full message as utf-8 if the message did not match a regex. It's kind of sloppy in the way that academic python often is, in ways that mostly don't matter because malformed data throwing an unhandled exception with a stack trace is just fine in academic research workflows.
But the change is inside an if/else block, so it only handles part of the bug. And the change simply sets a string equal to the utf-8 parse, so if that fails it will throw a different unhandled error that will propagate across the stack–it doesn't solve the issue and creates diagnostic weirdness, it's bad production code!
This is an unusually bad case. But it's not unique, and it highlights a variety of problems that are common across the dataset.
After expanding to a larger sample and seeing very low scores on reviews that seemed fine, I created an enhanced judge to analyze findings for (1) whether they were accurately grounded in the code, (2) whether the reviewer regarded them as blocking, (3) whether they were within the scope of the pr, vs observations of nearby pre-existing issues.
I then discovered that while the judge was reasonably consistent about matching findings to historical comments, it was not self-consistent at judging any of my new criteria, to say nothing of me agreeing with the verdicts. An important piece of this disagreement was the “blocking” assessment, where the source of truth is the reviewer and not the defect–if the code review is part of a CI system, the CI system determines whether the submission can merge. And advisory comments that don't block merge are appropriate for out of scope changes and nitpicks, while being absent from most historical PRs.
But since I couldn't automate reliable assessments of all that, I reduced scope and refocused.
A Better Ground Truth
The core problems I saw were:
Judge unreliability when identifying issues in prose reviews. Many review systems produce structured output; the judge should respect that structure if possible when enumerating findings.
Valid findings present in the code but unattested by human comments.
Valid comments on the code that are outside the scope of GitHub PRs. For example: flagging existing bugs, commenting on architectural choices, considering next steps or increased scope.
The first problem is easily solvable with a json specification delivered to the reviewer and parsed by the judge, with fallback to the preexisting prose judge if parsing fails.
The third problem is a filtering problem for my specific code review setup. Since I already established decent performance for my harness (54.5% recall on the WithMartian offline dataset; not published as it's unremarkable) messing around with filtering my findings to better match benchmark intention–such as by removing code comment corrections, architectural discussions, or options for future directions–wasn't appealing to me.
Scaling to the Full Dataset
The second problem is where the meat is. With frontier models finding bugs even in very well maintained code, it was implausible to me that LLM reviews are overwhelmed by false negative findings. Reviewing the full 1000 pr dataset with my system, which spends ~$25 of api-priced compute per review (somewhat more than predicted from early measurements), would be a $25,000 investment for a result I stopped caring deeply about. So instead, I had Claude haiku 4.5 review the full set. At closer to $0.10 per review I was able to run through this with my subscription at the discounted rate over time.
The haiku review run produced 2032 findings. I then ran the judging through haiku as well; judging runs are less expensive than review runs, but not enough so to make 1000 judge runs on opus affordable. I ran the findings through my v2 judge, first haiku-backed for price, to filter to only findings that were accurately grounded in the code being reviewed. Haiku filtered to 352 findings that were unattested, but which it believed were verifiably true within the code diffs.
Before the judging runs finished, I hand-evaluated 24 Haiku-backed findings, then evaluated all 24 with Opus. I chose three for promotion; Opus selected two of those three and one which I judged out of scope for the PR. This spot check satisfied me that Opus judgment was a sufficient filter for human attention.
Re-judging those 352 findings with Opus left me with 90 findings for manual human review. I evaluated all 90 by hand, looking at the judge assessment of the predicted finding, the submitted code diffs, and the later history of the PR before submission.
Results
Out of those 90 spot checked issues, I found 29 to be valid findings (in addition to eight findings I validated in early examples while evaluating the process). The most common way I established this was when the reviewer-predicted issue was fixed in a commit after the review window, but before the pr was merged. This is a clear and important data source for valid feedback, which is often not represented in comments; it is also an automatable pattern which can scale when building datasets from public repositories. Crucially, it can also scale with future commits to the repo, if flaws in code are revealed and fixed later in the repo history! It even has a weak negative signal–that if a a flaw in code was not revealed between its introduction and a refactor of that section, the risks of that flaw are sufficiently low.
Another common pattern was complaints about python 2 syntax being used in 10+ year old PRs. While I didn't promote these, scoring them as negatives for a coding agent in use today seems unsporting, at a minimum.
I also discovered two PRs in the dataset that were closed without being merged, and a third where the purpose of the PR was to quiet warnings about existing defects in the code–none of these are good fits for the situation being tested, and fill the benchmark metrics with noise.
The Bigger Problem
These are the same concerns that led OpenAI to create SWE-bench Verified; and also the same concerns that deprecated it. Point in time estimates don't capture a complete picture of truth or understanding, and manually reviewing these issues was a significant expense in both compute and human time.
In the case of SWRBench and WithMartian, I don't think that AI labs are fine tuning their models on benchmark performance. But they are doing RLVR and tuning on similar auto-graded datasets, and notably the hugging face incident was preceded by OpenAI training models to spend tokens on tasks that were unsolvable without hacking!
When models are in a situation where they have found the aligned answer (e.g. identified bugs that humans couldn't find, discovered that they lack the privileges to execute a task such as following links) but are still evaluated according to the flawed metric, we can only be optimizing them for misalignment. The fully general version of this argument is Goodhart’s law, but the in-practice mitigation for this is actually straightforward. Just like Goodhart’s law can often be solved in practice by using multivariate metrics and other simple strategies, bad benchmarks can be improved by iterated filtering and human validation-gated adjustments to the benchmark. I wonder if there is a BlockChain-style voting/verification algorithm that could be used to make these benchmarks living, iterable artifacts.
I want AI to be a technology that communicates with humans, robustly understands our goals and values, and cooperates with us to pursue and achieve those. That means being in conversation with the models in a much more interactive and nuanced way than many current benchmarks allow. And if we don't engage in that conversation, we are asking the models to guess and replicate our own limitations and biases, or to commit crimes on our behalf.