CWE-bench Reality Check: Defensive Cyber Patching, Cost, Hidden Tasks and Model-Harness Limits

Collinear AI released CWE-bench v0 in early September 2026 to measure a narrow but important capability: whether coding agents can audit real software for security weaknesses and produce patches that actually stop the exploit without breaking existing behavior. The benchmark is intentionally different from generic coding leaderboards. It gives an agent a repository and a security goal, withholds the vulnerability location, and asks the agent to find and remediate the weakness rather than merely reproduce a known patch.

The first results are notable, but they need careful interpretation. CWE-bench currently contains 100 permanently held-out audit-and-patch tasks spanning 54 CWE weakness types, six language buckets—C/C++, Go, Java, TypeScript/JavaScript, Python and Rust—and all ten OWASP 2025 categories. The held-out set is private to Collinear and Artificial Analysis, so outside organizations cannot download it.

What the current leaderboard says

On the current CWE-bench page, the highest deterministic pass@1 result is 47.8% for Claude Fable 5 using Claude Code, at an average billed API cost of $10.27 per rollout. Gemini 3.8 Flash Cyber using Antigravity is close at 47.2%, with a reported average cost of $3.64. GPT-5.6 Sol using Codex scores 44.2% at $2.29 per rollout. Other visible entries include Gemini 3.7 Flash at 44.0%/$1.43, Claude Opus 4.8 at 42.0%/$2.43, Grok 4.6 at 38.2%/$1.77 and Qwen3.8-Max at 37.5%/$1.45.

These are model-plus-harness results, not clean model-only measurements. Fable 5 is paired with Claude Code, Gemini variants with Antigravity, Sol with Codex, and several other models with opencode or their own CLI. Differences in repository navigation, tool use, prompting, retry behavior and context management can affect the result. It is therefore safer to say that these evaluated systems achieved the listed scores than to attribute every point of separation to the underlying foundation model.

The benchmark authors say the published v0 cost/performance results are averaged over four rollouts per task. The leaderboard score is the mean share of rollouts that pass the deterministic verifier across all 100 tasks. Average cost is mean billed API spend for one rollout on one task.

The benchmark tests remediation, not exploitation

CWE-bench is a defensive benchmark. Agents are asked to find and fix vulnerabilities; they are not tasked with creating exploits for deployment against real targets. Each task is based on a real open-source project pinned to a real commit, usually with a published advisory in its provenance. The benchmark then adds vulnerabilities of the same class that the upstream fix did not cover.

That construction is intended to reduce the value of memorizing a public advisory. The agent does not receive a CVE identifier, file hint or line number, and evaluation sandboxes disable internet access. A model that merely recognizes the original advisory and repeats its known patch can still fail because additional same-class weaknesses remain.

The public methodology describes four recurring failure modes: agents may fail to localize the vulnerable code, miss the security significance of code they inspect, patch the wrong invariant, or fix only some instances of a repeated weakness. This makes the benchmark useful for studying complete remediation rather than only bug recognition.

How grading works—and a documentation nuance

The current leaderboard states that its reported pass@1 score uses the deterministic programmatic verifier only. A rollout passes when the exploit no longer works and the pre-existing regression tests still pass. The site separately describes per-vulnerability judge scoring for partial credit and future rubric-based evaluation, but that judge score is not the gate used for the current leaderboard metric.

Collinear's launch blog describes the broader grading system as a combination of deterministic proof-of-concept checks and rubric-based checks. Read together, the two documents imply that rubric/judge information is available for analysis, while the headline leaderboard currently uses the all-or-nothing deterministic gate. That distinction matters when reproducing or comparing numbers.

There is also a small but meaningful documentation inconsistency. The current top banner says 18 of 100 tasks are unsolved by every tested model, while the FAQ lower on the same page says 10 of 100. The current banner and launch-era material support the 18 figure, but the disagreement means the exact unsolved count should be treated as documentation drift until Collinear publishes a clarified, versioned snapshot. It is not appropriate to silently choose one value and present it as uncontested.

Cost changes the practical ranking

Fable 5 has the highest displayed score, but Gemini 3.8 Flash Cyber is only 0.6 percentage points behind while its reported average rollout cost is much lower. GPT-5.6 Sol is another 3.0 points behind Fable 5 at a still lower average cost. That makes the result a cost-versus-success frontier rather than a simple winner-takes-all table.

The benchmark page does not publish confidence intervals for the gaps between the leading systems. With only a 0.6-point displayed difference between Fable 5 and Gemini 3.8 Flash Cyber, readers should not assume the ordering is statistically robust across reruns, harness updates or future task rotations without uncertainty estimates.

The right deployment metric is also not just cost per rollout. A team deciding between agents would want cost per verified successful remediation, total latency, number of retries, token usage, developer review time and false-confidence rate. CWE-bench exposes billed cost per attempt, but it does not yet provide a complete operational cost model.

Exact model names matter

The leaderboard currently names the leading Anthropic model Claude Fable 5. It does not say Fable 5.1. That distinction matters because later point releases can have different weights, safeguards or tool behavior. An article that silently relabels the result as Fable 5.1 would be inventing an evaluation that the benchmark does not currently publish.

Likewise, the current CWE-bench leaderboard does not show a GPT-6 Astra result. Astra may be stronger or weaker on this task family, but no exact result can be inferred from Terminal-Bench, CodeRabbit, SWE-bench or other unrelated evaluations. A future Astra row should be reported only when the benchmark operator publishes one with its exact harness and settings.

Gemini 3.8 Flash Cyber should also be kept separate from the general Gemini 3.8 Flash model. Google describes the Cyber variant as a cybersecurity-focused model with restricted access for trusted defenders. The CWE-bench row is for the Cyber variant paired with Antigravity, not a generic Gemini 3.8 Flash API result.

Held-out tasks reduce contamination but limit independent reproduction

CWE-bench's strongest methodological choice is also its biggest transparency tradeoff. The 100 evaluation tasks are permanently held out and unavailable outside Collinear and Artificial Analysis. This greatly reduces direct training contamination and prevents teams from tuning against the exact test set.

But a private benchmark cannot be independently reproduced task-for-task by the wider research community. External researchers can inspect the public methodology and leaderboard, but they cannot rerun the exact hidden set, independently audit every planted vulnerability, or verify sample-level scoring without cooperation from the benchmark holder.

That does not make the benchmark invalid. It means the evidence should be described precisely: it is a contamination-resistant, privately held evaluation with externally reported results, not a fully open reproducible benchmark.

CWE-bench is not SWE-bench Verified

SWE-bench Verified is a different evaluation. It consists of 500 human-validated software-engineering issues selected from the original SWE-bench test set. Those tasks come from 12 open-source Python repositories. Agents receive the GitHub issue statement and codebase, then must produce a patch that passes both FAIL_TO_PASS tests for the issue and PASS_TO_PASS regression tests.

OpenAI originally released Verified in 2024 after a large annotation campaign involving 93 software developers and 1,699 reviewed samples. In February 2026, however, OpenAI said it no longer considered SWE-bench Verified suitable for frontier launch evaluation because of benchmark contamination and test-quality problems. In its audit of a difficult subset, OpenAI reported that at least 59.4% of the audited problems had material test issues, and it recommended SWE-bench Pro instead.

A CWE-bench percentage therefore should not be compared directly with a SWE-bench Verified percentage. One tests security audit-and-remediation across multiple languages with hidden vulnerability locations; the other tests general issue resolution on a public, Python-only subset with a different harness and different sources of contamination.

CWE-bench is not SWE-bench Pro either

SWE-bench Pro is also a separate benchmark. Scale describes it as 1,865 software-engineering instances across 41 repositories: 731 public, 858 held-out and 276 commercial instances, drawn from 11 public, 12 held-out and 18 enterprise repositories.

Pro targets realistic general software-engineering work and uses a broader mixture of public, held-out and commercial code. CWE-bench instead focuses specifically on defensive security discovery and patching. Their task distributions, repository sets, harnesses and scoring criteria differ, so their raw percentages are not interchangeable rankings.

The useful comparison is methodological: both are attempts to make frontier coding evaluation harder to memorize and more representative of unseen work. CWE-bench pushes further toward a fully private security-specific holdout; SWE-bench Pro retains public and private components for a broader software-engineering scope.

What public feedback does—and does not—show

In this bounded verification pass, no sufficiently attributable first-hand X post or independent developer reproduction of CWE-bench was found that added reliable measured evidence beyond the benchmark operator's material. Generic launch reposts and promotional discussion are not a substitute for user testing.

That absence is itself worth recording. It would be misleading to claim that developers broadly agree with the ranking or that a social-media consensus has formed. The current evidence is primarily the benchmark operator's methodology and leaderboard, with the reported evaluation performed in collaboration with Artificial Analysis.

Practical takeaway

CWE-bench is one of the more interesting new coding-agent evaluations because it asks a difficult real-world question: can an agent discover and completely remediate security weaknesses when nobody tells it exactly where the bug is?

Its early results suggest that even the strongest evaluated systems still fail more than half of their deterministic audit-and-patch attempts, while cost varies substantially between near-frontier systems. The benchmark's private holdout is a sensible defense against contamination, but it also limits external reproduction, and the current documentation contains at least one unresolved count inconsistency.

For security teams, the best use of CWE-bench is as a specialized defensive-patching signal, not as a universal coding leaderboard, an offensive-cyber benchmark, or a replacement for SWE-bench Verified or SWE-bench Pro. The next evidence to watch is a versioned task snapshot, uncertainty estimates across repeated runs, clarified 18-versus-10 unsolved counts, tighter harness controls, and fresh evaluations of exact newer model versions such as Fable 5.1 or GPT-6 Astra if and when those results are actually published.