Amazon Releases BenchProofer to Verify Coding Benchmarks
Amazon researchers said in a blog post on Nov. 16 that they have released BenchProofer, an open-source tool that adds formal proofs to coding benchmarks. They also released SWE-Proof, a version of all 500 tasks in the SWE-Bench Verified dataset with those proofs attached.
The problem is simple. Coding agents are usually scored by running tests on the fixes they write. A fix can pass every test and still fail on an input nobody checked. Amazon also says agents can learn to game the tests. That matters because companies use benchmark results to make decisions, and some use them as the reward signal to train agents.
For each task, BenchProofer uses the known correct fix to build three things: a precise specification of what a right answer must do, an implementation that a verifier can read, and a proof that the two match. A verifier then checks the proof automatically. Amazon says the proof holds under stated assumptions. For existing code that a fix calls, the tool writes short contracts that it tests but does not formally verify.
Amazon ran the tool on two models, Claude Opus 4.8 and GPT-5.5. You can find both in our list of AI models. In audit mode, the extra checks flagged nearly one in three of Opus 4.8's test-passing patches because their behavior differed from the reference fix. In a sample reviewed by humans, 21 of 41 flagged patches had real bugs. When agents were given the specifications, their verified pass rates rose from 84.6% to 93.6% for Opus 4.8 and from 79.6% to 94.4% for GPT-5.5.
For people who use AI coding tools, the point is trust. A high score on a test-based benchmark may overstate how often a tool's code is truly correct. Test-based scoring is not unique to code, either. A recent benchmark covers professional spreadsheet work in a similar way.
Several things are unclear. The results come from Amazon's own study, and the source text does not say whether outside researchers have repeated them. The excerpt also does not give the standard SWE-Bench Verified scores for the two models, so the full size of the gap is unknown. Amazon says it released every gate verdict so anyone can recheck the work.
Sources
Cite this
Amazon Releases BenchProofer to Verify Coding Benchmarks, New Mind Museum, November 16, 2026. https://newmindmuseum.com/research/2026/11/benchproofer-amazon-verified-coding-benchmarks/
New Mind Museum. (2026, November 16). Amazon Releases BenchProofer to Verify Coding Benchmarks. https://newmindmuseum.com/research/2026/11/benchproofer-amazon-verified-coding-benchmarks/
“Amazon Releases BenchProofer to Verify Coding Benchmarks.” New Mind Museum, 16 Nov. 2026, https://newmindmuseum.com/research/2026/11/benchproofer-amazon-verified-coding-benchmarks/.
@misc{new-mind-museum-2026-amazon-releases-benchproofer-to-verify-c,
author = {{New Mind Museum}},
title = {Amazon Releases BenchProofer to Verify Coding Benchmarks},
year = {2026},
url = {https://newmindmuseum.com/research/2026/11/benchproofer-amazon-verified-coding-benchmarks/}
}Free to quote and cite with a link to this page.