🔍 Read the full analysis: CWE-bench V2: Building A More Transparent AI Benchmark on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Collinear AI released CWE-bench v2, a benchmark of 100 security-fix tasks from real open-source repositories, alongside its grader and evaluation harness. The leading reported result is 67.8% pass@1, but the published comparison does not show that any model solved every task, and key details needed to independently assess all results remain unclear.
Collinear AI has released CWE-bench v2, a public benchmark of 100 security-fix tasks drawn from real open-source repositories, along with a deterministic grader and evaluation harness, as detailed in the original analysis. The release reports results for 11 models, with Gemini 4 Argon leading at 67.8% pass@1; the benchmark is intended to let researchers compare coding agents on security repairs under a shared set of tasks.
Each task gives an agent a checkout of a real software repository and an instruction to find and repair a security weakness without breaking existing functionality. The tasks cover 91 CWE weakness categories. Collinear AI says the grader checks behavior programmatically, rather than requiring a particular code structure, to make repeated evaluations more consistent.
In the release’s reported results, Gemini 4 Argon scored 67.8% pass@1 and 81.0% pass@4, slightly ahead of Claude Opus 5.5 at 67.3% and 78.0%. Pass@1 records success on a single attempt; pass@4 allows up to four attempts. The release says at least one evaluated model solved 95 of the 100 tasks, but that figure does not mean one model completed all 95.
The authors also report cost comparisons. DeepSeek V4.1 Flash (1017) scored 57.0% pass@1 at a reported $0.04 per rollout; Claude Opus 5.5 scored about 10 percentage points higher at $1.83, while Gemini 4 Argon scored slightly higher than Claude at $3.92. These are figures from the release, and costs can vary with model pricing and evaluation conditions. The benchmark tasks are packaged as Harbor environments, with a registry file for running them through Hugging Face. Access to the dataset is gated and requires accepting its conditions.
What Security Repair Scores Show
The release offers a common, inspectable test for a difficult coding-agent task: fixing security weaknesses while keeping the surrounding software working. Public tasks and a shared grader can make it easier for teams to compare systems and examine why an agent passed or failed, rather than relying only on vendor claims or results from incompatible evaluations.
The scores also show why benchmark success should not be treated as proof that a model is ready to take responsibility for security work. The leading reported pass@1 result is below 70%, and the release says the leader fails 19% of tasks on every attempt. Real repositories differ in size, dependencies, test coverage and threat models. A score on this curated set cannot establish that an agent will identify or repair vulnerabilities safely across production systems.
The reported cost figures may help developers weigh performance against expense, but they are tied to the release’s evaluation setup. Without fuller information on configurations and pricing assumptions, they should not be treated as universal operating costs or a definitive ranking of value.
Products Worth Considering
Samsung Odyssey G4 25-inch FHD Gaming Monitor
Samsung 27-inch Odyssey G5 QHD Gaming Monitor (G51F)
Samsung 34-Inch Odyssey G5 Ultra-Wide Curved Gaming Monitor
As an affiliate, we earn on qualifying purchases.
How Version 2 Tests Repairs
Version 2 retains the benchmark’s task format and makes a 100-task set publicly available. According to Collinear AI, reviewers examined each task for realistic instructions and checked that the grading criteria could be inferred from those instructions. The authors say they verified that starter code fails the relevant checks, a reference fix passes, and existing functionality remains intact.
The release says verifiers were designed to reject memorized solutions, including known upstream patches, and to assess correct behavior rather than a single implementation. LLM judge panels helped shape programmatic checks for task difficulty and fairness, according to the authors. These are descriptions from the release; the supplied material does not include independent validation of every task or verifier.
Collinear AI also maintains a separate held-out set for tracking progress on tasks not seen by evaluated models. That set is not public, and teams must contact the company to request an evaluation. The release directs users to two Hugging Face channels: a dataset-page leaderboard for open-weight models and a leaderboard Space for broader comparisons, including API-only models.
“A defender has to anticipate the ways an attacker could get in, and get it right every single time.”
— Collinear AI, in its CWE-bench v2 announcement on Hugging Face
Limits of the Published Comparison
The supplied results do not provide enough methodological detail to independently assess every comparison. In particular, the exact model configurations and the conditions behind each cost estimate are not specified. Scores may also change as new submissions are added, and the release does not establish how closely performance on these 100 tasks predicts performance in production security work.
Because the held-out tasks are private, outside researchers cannot inspect or reproduce evaluations on that set. It is also unclear how performance varies across different repository types, vulnerability scenarios and deployment conditions. The public grader supports repeatable checks on the released tasks, but does not by itself resolve those broader questions.
More Runs and Task Feedback
Researchers and developers can evaluate models on the public set using Harbor after accepting the dataset’s access conditions and setting up Hugging Face authentication. Collinear AI invites feedback and task contributions; the release says leaderboard results can be submitted through pull requests.
Teams interested in the private held-out evaluation can contact Collinear AI. Further runs and outside review may add to the public comparisons and expose weaknesses in the tasks or grading checks. The release does not give a timetable for new results or specify when, or whether, the held-out set might become publicly available.
Key Questions
What is CWE-bench v2?
CWE-bench v2 is a benchmark of 100 tasks asking coding agents to repair security weaknesses in real open-source repositories while preserving existing behavior. The tasks span 91 CWE categories.
Which model had the highest reported score?
The release reports that Gemini 4 Argon led at 67.8% pass@1 and 81.0% pass@4, narrowly ahead of Claude Opus 5.5. These are the release’s results, and the supplied material does not give enough detail to independently assess every comparison.
Does the 95-task result mean one model solved 95 tasks?
No. The release says at least one evaluated model solved 95 of the 100 tasks; it does not say that a single model solved all 95. It also reports that the leading model failed 19% of the set on every attempt.
Can researchers run the benchmark themselves?
Yes. The public tasks are packaged as Harbor environments and distributed through Hugging Face. Users must accept the dataset’s access conditions and set up authentication before running them.
What is not public?
Collinear AI’s held-out evaluation set is private. The release says teams can contact the company to request an evaluation, but outside readers cannot directly inspect or reproduce results on those tasks.
Primary source: Hugging Face · via ThorstenMeyerAI.com
