Qwen3.8-27B Reality Check: 77.6 SWE-bench Verified Is Independent, 61.7 Pro Is Vendor-Run

What Qwen3.8-27B actually is

Alibaba's Qwen team released Qwen3.8-27B on August 14, 2026. It is an Apache-2.0 open-weight, dense 27B language model with a vision encoder rather than a small Mixture-of-Experts model. The official model card lists 64 language-model layers, Multi-Token Prediction (MTP), native image and video understanding, a 262,144-token native context window, and scaling support up to 1,000,000 tokens.

That combination makes Qwen3.8-27B unusually relevant to local and private deployments: it is much smaller than the largest frontier systems but is designed for coding, tool use, professional work and multimodal tasks. The important question is not whether one launch-table number looks impressive. It is which numbers have been reproduced, under what harness, and on what exact serving stack.

Primary sources:

• Official Qwen3.8-27B model card
• Official Qwen3.8 repository

SWE-bench Verified: 77.6% comes from a public community rerun

The strongest current independent coding evidence I found is not on Qwen's launch table. A public reproduction project running a RedHatAI/Qwen3.8-27B-NVFP4 serving route on two RTX 5090 32 GB GPUs reports 388 resolved tasks out of 500 on SWE-bench Verified, or 77.6%, on September 4.

The run is unusually well documented for a community benchmark:

• SWE-bench Verified: 388/500 = 77.6%;
• agent: mini-SWE-agent 2.4.6;
• harness: the official SWE-bench harness;
• attempts: one attempt per task;
• workers: 12;
• elapsed run time: about 3 hours 37 minutes;
• serving stack: a specific NVFP4 quantized Qwen3.8-27B route on two RTX 5090s.

The same project reports 386/500 = 77.2% on its previous FP8 serving shape under the same general harness. This is useful because the project publishes scripts, hardware details and result directories rather than only a screenshot.

But the qualification matters: 77.6% is evidence for this quantized deployment plus mini-SWE-agent scaffold, not a pristine BF16 model-only score. Agent scaffolding, quantization, runtime configuration, context handling and retries can move software-engineering results.

Source:

• Independent Qwen3.8-27B RTX 5090 benchmark project
• Detailed benchmark results

SWE-bench Pro: Qwen reports 61.7%, but it is a different evaluation

Qwen's own model card reports 61.7 on SWE-bench Pro. That number must stay separate from the 77.6 SWE-bench Verified result because these are different benchmark suites and different evaluation paths.

Qwen's footnote says its Pro evaluation used the Claude Code harness, temperature=1.0, top_p=0.95, and a 256K context window. It also says problematic tasks were corrected and the baseline models were re-evaluated on the refined benchmark.

There is another comparison caveat. The same table lists Claude Opus 4.6 Max at 53.4, but Qwen explicitly says that Opus row uses its officially reported score, while the other models were run through Qwen's evaluation setup. Therefore the table is not clean evidence that Qwen3.8-27B universally "beats Opus" on an identical harness and task revision.

I did not find a separate, same-checkpoint independent rerun that reproduces Qwen's 61.7 SWE-bench Pro result with the same corrected task set and Claude Code configuration. Until one appears, the fairest label is vendor-run, method-documented, not independently reproduced here.

The two software-engineering results should therefore be read as:

These percentages should not be subtracted from each other or used to rank the suites. Verified and Pro differ in task construction, difficulty, harness history and evaluation provenance.

The rest of Qwen's coding and agent table is promising, but still first-party

Qwen reports 73.0 on Terminal-Bench 2.1, 42.3 on NL2Repo-Bench, 42.2 on DeepSWE 1.1, 90.3 on LiveCodeBench v6, and 20.4 Pass@1 / 42.9 score on Agents' Last Exam.

Those numbers help characterize the model, but they are not interchangeable. Terminal work, repository generation, issue resolution, competitive coding and long-horizon agent tasks measure different capabilities. Qwen also labels QwenSWEBench and CoWorkBench as in-house benchmarks, so those should not be treated as independent confirmation.

A useful sign of restraint is that the official table does not provide a Qwen3.8-27B SWE-bench Verified row. The 77.6 Verified result above comes from the separate community reproduction, not from silently converting Qwen's Pro result into Verified.

Multimodal results are strong on the vendor table, not independently settled

Qwen3.8-27B is a native vision-language model. Qwen reports 84.3 on OSWorld-Verified, 64.8 on WebArena-Verified, 81.9 on AndroidWorld, 38.6 on SWE-MM, and 62.9 on Vision2Web.

The methodology again matters. Qwen says WebArena-Verified uses the official grader under the OSWorld scaffold; SWE-MM uses the public development split with modifications described in the Claude Opus 4.7 system card; Vision2Web uses the Claude Code harness and a GPT-5.4 snapshot as judge. Several visual reasoning datasets also use manually corrected annotations.

These are useful first-party measurements, not one universal "multimodal score." I did not find a same-checkpoint independent rerun covering this full multimodal table under identical settings.

Artificial Analysis currently gives a very different kind of score: 34

Artificial Analysis currently reports 34 on Intelligence Index v4.3 for Qwen3.8-27B at xhigh reasoning. That composite is completely separate from SWE-bench.

Its current model page also reports approximately:

• 47.5 output tokens/s on the measured Alibaba API route;
• 3.83 seconds time to first token;
• $0.50 per million input tokens and $3.00 per million output tokens on that route;
• $0.82 cost per Intelligence Index task;
• roughly 200 million output tokens consumed across the Intelligence Index evaluation.

Artificial Analysis v4.3 combines multiple evaluation families including AA-Briefcase, GDPval-AA v2, AutomationBench-AA, Terminal-Bench v4.0, SciCode, Humanity's Last Exam, GDP.pdf, CritPt, AA-Omniscience and its long-context reasoning benchmark. A score of 34 on that composite cannot be compared numerically with 77.6 Verified or 61.7 Pro.

The page is also dynamic. Provider speed, latency, pricing and even benchmark snapshots can change, so these figures are best treated as a point-in-time independent measurement, not a service-level guarantee.

Source:

• Artificial Analysis: Qwen3.8-27B

Local performance varies enormously with hardware, quantization and runtime

The public evidence also explains why there is no honest single "Qwen3.8-27B local speed."

The two-RTX-5090 benchmark project has spent multiple iterations tuning vLLM, NVFP4 KV cache, MTP and communication kernels. A separate September 5 community test on an Intel Arc Pro B70 32 GB with GPTQ INT4, BF16 MTP4 and vLLM XPU reports median decode throughput falling from 81.24 tok/s at a 512-token prompt to 50.31 tok/s at 120K prompt length.

At the other end, a September 2 RTX 5080 16 GB setup using a Q4_K_M quant with partial CPU offload reports roughly 13.25 tok/s around 50K context and 13.06 tok/s around 61K. Those are not contradictory results: they use different GPUs, quantizations, runtimes, offload strategies and context lengths.

This hardware spread is more useful than a single viral speed claim. Dense 27B inference can be fast on tuned hardware, but memory bandwidth, KV-cache format, speculative decoding, context length and CPU offload substantially affect throughput.

Community measurements:

• Intel Arc Pro B70 context-scaling test
• RTX 5080 16 GB Q4_K_M tuning test

Practitioner feedback is positive on capability but divided on reasoning style and knowledge

Public feedback is not a controlled benchmark, but it can reveal practical failure modes worth testing.

Simon Willison's August 16 hands-on report found the model capable at local coding, tool use and vision tasks, but criticized the default xhigh reasoning setting for excessive deliberation and slow responses. He reported much faster turnaround after lowering or disabling reasoning, with some loss in task quality on his examples.

A separate LocalLLaMA discussion reported weaker obscure factual recall than Qwen3.6 in the author's personal offline tests, while other commenters in the same thread said they saw a similar tradeoff but still liked Qwen3.8's coding and tool use. Another recent thread praised a 24 GB local deployment for long-running agentic coding.

These are self-selected anecdotes, not evidence of a population-wide consensus. They do, however, align with two practical tests a prospective user should run: measure real task completion at different reasoning-effort levels, and test offline factual recall separately from tool-assisted research.

Sources:

• Simon Willison's August 16 hands-on report
• LocalLLaMA knowledge-regression discussion
• LocalLLaMA agentic-coding discussion

I searched for a stable, directly attributable Qwen launch post or controlled Qwen3.8-27B rerun on X in this pass. Search surfaced X trend summaries rather than a reliable direct post suitable for citation. I therefore do not infer an X consensus or quote an unattributed reaction.

Access, context and practical tradeoffs

The official weights are Apache-2.0 and support self-hosting through Transformers, vLLM, SGLang and other runtimes. The model card lists 262,144 native context and extension up to 1M tokens. Qwen also supports adjustable reasoning effort (xhigh, medium, low) and preserves prior reasoning state by default in supported workflows.

The official card itself warns that lower reasoning effort does not always reduce total agent-task latency: weaker per-turn analysis can cause more failures and retries. That is a useful reminder that tokens-per-second is not the same as end-to-end productivity.

For cloud use, current independent measurement from Artificial Analysis lists the Alibaba route at $0.50/M input and $3.00/M output. For local use, the economic calculation changes to hardware acquisition, electricity, memory capacity and engineering time. A "free local model" still has infrastructure cost.

Evidence level and practical verdict

Qwen3.8-27B has stronger evidence than a typical launch-table-only release because a public community project has now produced a full 500-task SWE-bench Verified run with scripts, harness details, hardware and result directories. That makes the 77.6 Verified result meaningful.

It does not validate every other Qwen claim. The 61.7 SWE-bench Pro score remains first-party under a refined task set and Qwen's Claude Code setup; the multimodal scores are primarily vendor-run; Artificial Analysis uses a different composite; local throughput depends heavily on the serving stack.

The most defensible conclusion today is therefore narrower than "Qwen3.8-27B beats frontier models." It is an unusually capable open-weight 27B multimodal model with a strong, documented independent SWE-bench Verified result and credible local-agent use cases. Its biggest comparison risks are benchmark mixing, quantization/harness differences, reasoning verbosity and hardware-sensitive latency.

The next evidence that would materially change this assessment is a same-checkpoint independent SWE-bench Pro rerun with the exact task revision and harness disclosed, plus matched BF16-versus-quantized testing and independent multimodal-agent reruns.