MiniCPM5-2B Reality Check: 46.4 SWE-bench Verified Is Vendor-Reproduced; Current AA Index Is 13, and Sampling Can Matter
OpenBMB's 2.5B dense model is unusually strong for its size, but its 46.4 SWE-bench Verified and 14.4 SWE-bench Pro scores are vendor-reproduced. We separate current AA v4.3 evidence from launch claims and inspect a new sampling-sensitivity report.
What shipped
OpenBMB released MiniCPM5-2B on September 7, 2026 as the second MiniCPM5 model. The final checkpoint is a dense text-only causal language model using the standard LlamaForCausalLM architecture, with 2,516,756,480 total parameters, 1,981,982,720 non-embedding parameters, 42 layers, grouped-query attention with 16 query heads and 2 KV heads, and a 131,072-token context window. The weights and repository are released under Apache-2.0.
Official model card:
The release is unusually developer-friendly for a small model. OpenBMB publishes the final RL+OPD checkpoint plus SFT, mid-training and base checkpoints, and separate GGUF, MLX, GPTQ and speculative-decoding artifacts. The model card documents Transformers, vLLM, SGLang, llama.cpp, Ollama, LM Studio and MLX paths, with SGLang recommended for its XML-style tool calls.
That makes MiniCPM5-2B interesting for local assistants, coding agents and tool-use experiments. It does not make every launch benchmark independently verified.
The headline 53.9 average is a vendor comparison
OpenBMB says MiniCPM5-2B averages 53.9 across its published comparison table and exceeds the highest larger model included in that selected table, Qwen3.5-4B at 51.1. The same table reports strong results in coding, mathematics, long-context, tool use and agent tasks.
The provenance note matters. OpenBMB explicitly says scores marked with a dagger come from the official Artificial Analysis release; all other rows were reproduced internally. Therefore the attention-getting results such as 69.1 LiveCodeBench v6, 86.5 AIME 2025, 66.6 BFCL v4, 88.7 GAIA Text-103, 46.4 SWE-bench Verified and 14.4 SWE-bench Pro should be described as OpenBMB-reproduced results unless and until an independent evaluator publishes a pinned reproduction.
The card does not provide a complete per-benchmark disclosure of task counts, exact harness versions, retry/pass policy, decoding settings and run dates for every internally reproduced row. That limits how precisely these numbers can be compared with another lab's score even when the benchmark name is the same.
SWE-bench Verified and SWE-bench Pro must stay separate
The model card reports:
- SWE-bench Verified: 46.4
- SWE-bench Pro: 14.4
Those are very different task sets and difficulty regimes. They should never be merged into one generic “SWE-bench score.”
The 46.4 Verified result is especially striking for a roughly 2.5B-parameter dense model. It is also not marked with the Artificial Analysis dagger, so it belongs in the vendor-reproduced evidence column, not the independent-results column.
The 14.4 SWE-bench Pro result tells a more cautious story. In OpenBMB's own comparison table, Qwen3.5-4B scores 28.2 on SWE-bench Pro. That does not invalidate MiniCPM5-2B's coding strengths; it shows why a single favorable coding benchmark should not be used to claim universal coding leadership.
A rigorous reproduction should pin the exact SWE-bench Verified or SWE-bench Pro revision, agent scaffold, repository setup, tool access, reasoning mode, token/time budget, retries and pass definition.
Terminal-Bench 2.1 is not Terminal-Bench 4.0
OpenBMB lists 8.6 on Terminal-Bench v2.1, and that row is dagger-marked as coming from Artificial Analysis rather than OpenBMB's internal reproduction.
Artificial Analysis has since moved its Intelligence Index to v4.3, whose benchmark bundle now includes Terminal-Bench v4.0, not v2.1. Its current MiniCPM5-2B page reports an overall Artificial Analysis Intelligence Index score of 13.
Current independent reference:
The correct interpretation is not that an 8.6 Terminal-Bench 2.1 score can be compared numerically with a Terminal-Bench 4.0 score. The benchmark revision changed, and the current Artificial Analysis page does not expose a MiniCPM5-2B Terminal-Bench 4.0 number in the reviewed public text. Until an exact v4.0 row is available, it should remain unknown, not inferred.
Why you may see different Artificial Analysis headline numbers
Some launch-day posts and partner pages cite higher Artificial Analysis index figures for MiniCPM5-2B. The current Artificial Analysis model page, verified on September 9, reports 13 on Intelligence Index v4.3 and ranks the model first in its tiny open-weight comparison class on that page.
That is the number to use for the current v4.3 snapshot. Older index values can reflect a different index revision and benchmark composition. They should be dated and versioned rather than presented as if the model's capability changed under an identical test.
Artificial Analysis also shows no standardized output-speed result for MiniCPM5-2B on the current page. For a self-hosted open-weight model, latency and tokens per second depend heavily on hardware, quantization, runtime, context length and batch/concurrency settings. A general “fast” claim without a pinned hardware-and-software setup is not a reproducible latency benchmark.
“$0 per million tokens” does not mean inference is free
Artificial Analysis displays $0/M input and $0/M output for MiniCPM5-2B because it is an open-weight self-hostable model rather than a metered first-party API product in that comparison.
That is a model-price representation, not total cost of ownership. Local or hosted deployment still incurs hardware, electricity, cloud GPU/NPU, memory, engineering and operations costs. There is no single universal per-token price for self-hosting.
For practical buyers, the more useful comparison is cost per successful workload on a declared device or provider, including retries and latency—not a zero token-license price by itself.
Training openness is meaningful, but it does not remove evaluation uncertainty
OpenBMB released substantial training artifacts alongside the model. The model card points to UltraX and UltraData-Code, a 500K-sample agent SFT dataset, and 80K+ RL samples covering mathematics, code, general knowledge and long-context reasoning.
OpenBMB says the post-training pipeline uses SFT, reinforcement learning and On-Policy Distillation (OPD). It reports that RL+OPD adds an average 10.96 points on selected reasoning/general benchmarks and 6.96 points on selected agentic benchmarks, with OPD merging capabilities from 16 RL expert models, including 5 agentic experts.
Those ablation claims are useful evidence about the intended training recipe, but they are still producer-run measurements. Releasing data and checkpoints improves inspectability; it does not automatically make the benchmark results independent.
A fresh community report shows why inference settings matter
A public GitHub issue opened on September 8 reports a potentially important deployment sensitivity. The author tested the official Q8_0 and Q4_K_M GGUFs on an RTX 3060 12GB with llama.cpp, thinking enabled, and HumanEval+ over 164 tasks. The report says the documented sampling settings—temperature 1.0 and top-p 0.95 without an explicit repeat penalty—produced frequent repetitive “runaway” generations in that setup.
The issue reports that adding a repeat penalty materially changed both the measured score and the author's custom runaway rate, with the best tested point at 1.15. The author also clearly states that the harness uses one repair round, so the absolute scores are not directly comparable to standard pass@1, and that “runaway rate” is their own definition.
Primary discussion:
This is valuable user evidence because the setup, hardware, quantizations and caveats are disclosed. It is still one user's measurement on one machine with one harness. It should not be generalized into a claim that all MiniCPM5-2B deployments require a specific repeat penalty.
OpenBMB's official card currently recommends temperature=1.0, top_p=0.95. The GitHub issue is open at verification time, with no maintainer response captured in the reviewed issue. That makes the sampling question a reproduction target, not a settled best practice.
Public reaction is positive but too thin for a consensus claim
An accessible Reddit launch thread praised the model's size-to-capability ratio and repeated the sub-4B benchmark story, with commenters expressing surprise and enthusiasm. The thread is small and self-selected, and the headline largely echoes launch material.
Discussion:
That is anecdotal community reaction, not a representative user survey or independent benchmark. The GitHub issue above is more technically informative because it includes a reproducible-looking configuration and measured failure mode, while still needing replication.
Practical tradeoffs
MiniCPM5-2B's strongest verified attractions are concrete: Apache-2.0 open weights, a compact dense architecture, a 131K context window, broad local-runtime support, released training artifacts, tool-calling support and an independently tracked Artificial Analysis v4.3 score that is unusually strong for its size class.
The main reasons for caution are equally concrete:
- Many of the most impressive launch scores, including SWE-bench Verified 46.4 and SWE-bench Pro 14.4, are internally reproduced by OpenBMB.
- The Terminal-Bench 8.6 number is for v2.1, while current Artificial Analysis uses v4.0 in its composite.
- The current independent AA page reports Index 13, so older launch-day AA figures need explicit version labels.
- No standardized independent speed measurement is currently shown on the AA model page.
- Self-hosting has real infrastructure cost even when model-token pricing is shown as zero.
- A fresh one-user GGUF test suggests sampling/quantization settings can materially change local coding behavior, but that result has not yet been independently replicated.
Bottom line
MiniCPM5-2B is a credible and unusually capable-looking small open-weight release, but its most eye-catching coding and agent numbers need provenance labels.
The clean evidence table today is: 46.4 SWE-bench Verified — OpenBMB-reproduced; 14.4 SWE-bench Pro — OpenBMB-reproduced; 8.6 Terminal-Bench 2.1 — daggered to Artificial Analysis in the model card; current Artificial Analysis Intelligence Index v4.3 — 13; exact current Terminal-Bench 4.0 score — not established in the reviewed public page.
For real deployment decisions, the next high-value evidence is not another aggregate leaderboard. It is independent reproduction of the two SWE-bench variants, a pinned Terminal-Bench 4.0 result, hardware-normalized latency/throughput, and replication of the reported sampling sensitivity across quantizations and runtimes.
This article is built from the source material below. Open the originals for full context and the latest updates.
