SWE-bench Multimodal v2 Is Open Source: What the 480-Task Benchmark Actually Measures

The September 1 update is about reproducibility, not a new model leaderboard

The SWE-bench team released SWE-bench Multimodal v2 on September 1, 2026 and says the final v2 test split contains 480 tasks available for local evaluation. The release focuses on making visual software-engineering evaluation more reproducible: known flaky or ungradeable cases were removed, Docker environments were rebuilt, and JavaScript grading plus visual test-asset handling were hardened.

That makes v2 a meaningful benchmark update even without a new headline model ranking. It is designed to test whether an AI coding system can understand a real software issue that contains visual evidence, inspect the repository, produce a patch, and pass the repository's tests.

Primary sources: SWE-bench Multimodal, SWE-bench GitHub repository, and the v2 task changelog.

What makes Multimodal different

The original SWE-bench benchmark is centered on real GitHub issues and code changes. SWE-bench Multimodal shifts the domain toward visual, user-facing software. The associated dataset describes JavaScript and TypeScript repositories where tasks include image assets such as screenshots, screen recordings, diagrams, or renderings of incorrect output.

The currently published dataset card documents 17 JavaScript/TypeScript repositories across development and test splits. It describes a 510-instance historical test split and a 102-instance development split; those counts reflect the pre-v2 dataset card rather than the final v2 test split. The v2 release page and task changelog are the authoritative sources for the current 480-task test split.

This version distinction matters. A benchmark name without a dataset version is not enough provenance for a modern leaderboard result.

Dataset source: SWE-bench Multimodal on Hugging Face.

How 510 historical test tasks became 480 v2 tasks

The public repair log makes the curation process unusually inspectable. On August 11, the v2 work reduced the historical 510-task test split to 485 after removing cases that could not be made reproducible. Subsequent repairs and deprecations reduced it further. The changelog records, for example, two OpenLayers rendering instances retired because their visual baselines depended on GPU/driver behavior, a bpmn-js task whose gold patch still failed its graded tests, Quarto tasks retired because their evaluation stack was unusually slow and fragile, and an Alibaba Fusion task deprecated after an intermittent hang that could occupy a worker indefinitely. By August 18, the changelog records the test split at 480, matching the September 1 release page.

The important point is not simply that 30 tasks disappeared. The benchmark team published why problematic tasks were removed and kept deprecated material separately rather than silently scoring unstable cases.

Source: SWE-bench Multimodal task changelog.

The harness changes are part of the benchmark

Multimodal coding evaluations are especially sensitive to environment drift. The v2 changelog documents browser sandbox problems on modern Docker, non-UTF-8 test output that could discard an evaluation, patch-application fallbacks contaminating later attempts, Chrome/WebGL differences affecting rendering tests, dependency-download behavior, and visual assets that previously depended on external hosts.

The team rebuilt Docker environments and mirrored many binary test assets. It also changed repository cloning to avoid packing unnecessary history into evaluation images. One documented Carbon image set fell from roughly 9.9 GB compressed to about 1.53 GB average per image after the cloning fix; the changelog says this cut a cold full-split run from more than ten hours to about 5.3 hours in that repair exercise.

These details are not trivia. For an agent benchmark, the model, scaffold, environment, browser, test harness, concurrency, and dataset revision can all affect the measured resolve rate. A score without these details has lower evidentiary value.

Do not call old Multimodal scores 'v2' without v2 provenance

As of this verification pass, the official September 1 v2 release page announces the 480-task open-source split and tooling but does not publish a new v2 model leaderboard table on that page. Third-party benchmark aggregators currently show model scores under the broader label “SWE-bench Multimodal,” including recent frontier-model rows, but those pages do not consistently identify a 480-task v2 run, task manifest, harness commit, or evaluation trajectory.

For that reason, this article does not relabel those scores as SWE-bench Multimodal v2 results. A defensible v2 comparison should show, at minimum, the exact 480-task dataset revision, model identifier, agent/scaffold, tool permissions, evaluation harness version, run date, and resolved-task count. Until that provenance is available, a 2026 score carrying only the generic “SWE-bench Multimodal” name should be treated as version-ambiguous.

This is also why a reported Claude Fable 5.1 Multimodal score, for example, should not be presented here as a v2 score merely because it appears on a current aggregator page.

SWE-bench Multimodal v2 is not SWE-bench Verified

SWE-bench Verified is a different benchmark slice. It contains 500 human-validated instances from the original SWE-bench and focuses on software issues whose problem statements and tests were reviewed for solvability and quality. The official SWE-bench site also maintains a bash-only comparison lane using mini-SWE-agent to make language-model comparisons more controlled.

Verified should therefore remain its own column in any model comparison. Its 500 tasks are not the 480 visual JavaScript/TypeScript tasks in Multimodal v2, and a model's score on one cannot be substituted for the other.

OpenAI has also argued in 2026 that SWE-bench Verified no longer cleanly measures frontier coding capability because of contamination and residual task-quality problems. That criticism is relevant context, but it does not transform Multimodal v2 into a replacement for Verified; they test different slices of software-engineering behavior.

Sources: SWE-bench Verified and OpenAI's 2026 Verified audit.

SWE-bench Pro is a third separate lane

SWE-bench Pro, maintained by Scale, is also not directly interchangeable with Multimodal v2. Scale documents 1,865 total tasks across 41 professional repositories: 731 public instances, 276 private instances, and 858 held-out instances. Its methodology emphasizes longer-horizon professional codebase work, contamination resistance, Docker environments, human-augmented problem specifications, and fail-to-pass plus pass-to-pass testing.

That makes Pro useful for a different question: how well an agent handles complex repository-level software work across public and private codebases. Multimodal v2 asks a narrower but important question about visual software domains and JavaScript/TypeScript issue resolution.

A fair model report should keep at least three separate lines: SWE-bench Verified, SWE-bench Pro, and SWE-bench Multimodal v2. It should not average or rank them as if they were the same test.

Source: Scale SWE-bench Pro methodology.

What developers can reproduce now

The SWE-bench repository supports multimodal as an evaluation alias and uses Docker-based environments. The official README documents local evaluation commands and says evaluation summaries are written to the run logs. The v2 release also states that the complete test split and evaluation tooling are open source.

That matters for evidence quality. A vendor can publish a benchmark number, but an independently reproducible run with a pinned model endpoint, scaffold, dataset commit, harness commit, and task-level results is stronger evidence. The open v2 release makes that kind of verification more practical.

There are still limitations. Some tasks are browser-heavy, image assets can be large, and the changelog records remaining known issues, including a small set of Carbon accessibility tests that can depend on a network-fetched rule engine. The benchmark is therefore more reproducible than before, not magically free of infrastructure sensitivity.

What v2 tells us about model capability

Multimodal v2 is valuable because text-only repository benchmarks can miss a common software-development reality: bug reports often arrive with screenshots, broken layouts, diagrams, maps, charts, or other visual evidence. A capable coding agent needs to connect that visual information to source code and tests.

But a high Multimodal score would still not prove universal coding superiority. The benchmark is concentrated in JavaScript/TypeScript and visual software domains. It does not directly measure every backend language, mobile stack, infrastructure workflow, security task, or greenfield architecture problem. It also measures an agent system, not just isolated next-token model quality; scaffold and tool choices matter.

The right interpretation is therefore task-specific: Multimodal v2 can strengthen evidence about visual issue resolution and cross-modal software reasoning when the run is versioned and reproducible.

Public feedback is still too sparse for a sentiment claim

I did not find a sufficiently attributable X thread or a substantial public developer discussion specifically evaluating the final 480-task v2 release during this pass. Search results mostly point back to the official release and generic leaderboard pages. That is not enough to claim developer consensus, positive or negative.

The public task changelog does provide unusually detailed engineering evidence about what maintainers encountered while stabilizing the benchmark, but those maintenance notes are not community sentiment. Until reproducible third-party v2 runs and attributable practitioner reports appear, measured results should carry more weight than social impressions.

Practical checklist for reading the next v2 score

When a lab or evaluator publishes a SWE-bench Multimodal v2 number, check six things before comparing it: the exact dataset/task revision, exact model version, agent scaffold, allowed tools and image handling, harness/container revision, and resolved-task count with run date. If any of those are missing, treat the score as lower-confidence.

Most importantly, do not mix a generic pre-v2 Multimodal number with the final 480-task v2 split, and do not compare it directly with SWE-bench Verified or SWE-bench Pro as if all three were one leaderboard.

Confidence is high on the v2 release date, 480-task final split and repair history because they come from the benchmark team's own site and repositories. Confidence is medium on broad performance implications because same-harness frontier-model v2 results with complete public provenance are not yet established in the sources reviewed here.