MiniCPM5-2B Reality Check: 23→15→14 Is an Index Revision; SWE-bench Verified 46.4 and Pro 14.4 Are Vendor-Run

OpenBMB released MiniCPM5-2B on September 7, 2026 as a compact, text-only reasoning model aimed at local assistants, coding agents, tool use and resource-constrained deployment. The official MiniCPM5-2B model card lists 2,516,756,480 total parameters, 1,981,982,720 non-embedding parameters, 42 layers, grouped-query attention with 16 query heads and 2 KV heads, and a 131,072-token native context window. The weights are released under Apache 2.0, so the practical access story is primarily download-and-self-host rather than a single mandatory hosted API.

The model is interesting because its size-to-capability results are unusually strong, but several numbers circulating around launch day come from different benchmark versions or different evaluators. They should not be merged into one leaderboard story.

The 23 → 15 → 14 sequence is mostly benchmark-version drift, not evidence of a sudden collapse

OpenBMB's September 7 launch post on X promoted an Artificial Analysis Intelligence Index score of 23. Artificial Analysis subsequently explained in its MiniCPM5-2B release evaluation that 23 was the score under Intelligence Index v4.1.1. Under the revised v4.2 evaluation mix and weighting, Artificial Analysis measured 15 and explicitly warned that scores across those two index versions are not directly comparable.

Artificial Analysis changed the index again on September 7. Its v4.3 methodology update replaced Terminal-Bench 2.1 with the harder Terminal-Bench 4.0 and replaced τ³-Banking with the private-set AutomationBench-AA. The current MiniCPM5-2B model page shows 14 on v4.3, but labels that value an estimate with independent evaluation forthcoming.

So 23, 15 and 14 are not three clean measurements of the same test. The strongest defensible interpretation is: MiniCPM5-2B remains unusually competitive for a ~2.6B dense open-weight model, while the composite index moved underneath it. Treating 23→14 as a 39% capability loss would be misleading.

SWE-bench Verified and SWE-bench Pro must be kept separate

OpenBMB's model card reports:

• SWE-bench Verified: 46.4
• SWE-bench Pro: 14.4

These are materially different benchmarks and should never be blended. The model card's notation matters too: OpenBMB says scores marked with a dagger (†) come from the official Artificial Analysis release, while unmarked scores are reproduced internally. The 46.4 Verified and 14.4 Pro rows are unmarked, so they should be treated as OpenBMB-run results, not independent reproductions.

The same card does not provide enough run-level detail beside those rows to reconstruct a matched external evaluation: it does not state the exact agent harness, task subset/count, retry policy, per-task outputs, environment image, or confidence interval for the two SWE-bench rows. That does not make the scores invalid, but it lowers confidence in cross-vendor ranking until an independent evaluator publishes a pinned, matched run.

This distinction is especially important because SWE-bench Verified has become relatively saturated for frontier systems, while SWE-bench Pro is intentionally more difficult. A 46.4 Verified result and 14.4 Pro result describe different difficulty regimes rather than a contradiction.

Terminal-Bench 2.1 is independent here; Terminal-Bench 4.0 is a different test

For Terminal-Bench v2.1, OpenBMB's table gives 8.6†, and Artificial Analysis's September 7 release article rounds the result to about 9%. Because the dagger identifies Artificial Analysis as the source, this is stronger independent evidence than the vendor-run SWE-bench rows.

But that 8.6/9% result should not be compared directly with current Terminal-Bench 4.0 scores. Artificial Analysis describes Terminal-Bench v4.0 as a harder 66-task benchmark with recalibrated compute/time allowances, improved environments and verifiers, and pass@1 averaged over three repeats per task. Its v4.3 methodology uses a mini-SWE-agent harness for the 4.0 evaluation.

At verification time, Artificial Analysis had not published a completed independent MiniCPM5-2B Terminal-Bench 4.0 score; the model's overall v4.3 index is still marked estimated. Therefore this article does not invent a 4.0 result or rank 8.6 from v2.1 against current 4.0 leaders.

OpenBMB's 53.9 average is useful, but it is not a universal leaderboard

OpenBMB reports an average of 53.9 across its selected comparison table, ahead of Qwen3.5-4B at 51.1 within that table. The individual rows span very different tasks: for example, the card reports 69.1 on LiveCodeBench v6, 86.5 on AIME 2025, 86.5 on AIME 2026, 68.1 on NoLiMa, 97.1 on τ²-Bench Telecom, and 66.6 on BFCL v4.

An arithmetic average across heterogeneous benchmarks is best read as a vendor summary of its chosen suite, not as a standardized universal score. Different tests have different scales, contamination risks, harnesses and failure modes. The more useful signals are the individual benchmark names, versions, evaluator provenance and run methodology.

Artificial Analysis provides a valuable independent counterweight. In its v4.2 evaluation it measured MiniCPM5-2B at GDPval-AA v2 Elo 831, τ³-Banking 21%, and AA-Briefcase Elo 438. It also noted a revealing behavior on AA-Omniscience: the model attempted only 29% of questions, reached 78% non-hallucination, but had only 8% raw accuracy. That means abstention behavior materially affects aggregate interpretations.

Context, access, pricing and latency tradeoffs

The model is text-only, has a 131,072-token native context window, and ships as open weights under Apache 2.0. OpenBMB documents support paths through Transformers, vLLM, SGLang and other local runtimes, including a speculative-decoding draft model.

There is no single official retail API price that can be used as "the price of MiniCPM5-2B." Self-hosting cost depends on hardware, quantization, context length, batching and runtime. Artificial Analysis's current page shows hosted speed/cost fields as unavailable for this model; its displayed zero-dollar token fields should not be interpreted as proof of a free production API.

Likewise, there is no standardized, independently reproduced p50/p95 latency figure in the reviewed release material. Local users can see very different throughput depending on hardware and quantization, so isolated tokens-per-second anecdotes are not equivalent to an API latency benchmark.

Public feedback is mixed and highly selection-biased

Launch-day discussion provides useful practical clues but not benchmark-grade evidence. In a September 7 r/LocalLLaMA release thread, users showed strong interest in using the model for local assistant pipelines and small agent workloads. In a separate September 7 r/LocalLLM thread, one commenter worried that the model might be "benchmaxxed," while a MacBook Air M1 user later described it as fast and surprisingly capable.

Other first-person reports disagree. A September 8 r/LocalLLaMA discussion described a Q8 CPU run around 6 tokens/s but said the model made enough mistakes that a much larger quantized MoE was preferred despite lower speed. These posts use different hardware, quantizations, prompts and workloads, so they should be treated as anecdotes rather than a measured community consensus.

The exact OpenBMB X launch post is included above because it is the primary public claim being audited. Direct X retrieval can be access-restricted for automated crawlers; Artificial Analysis independently confirms that the launch-day 23 corresponded to v4.1.1, which is the key fact needed to interpret that post fairly.

Bottom line

MiniCPM5-2B is a notable compact open-weight release: 2.52B parameters, 131K context, Apache 2.0 licensing, and strong coding/tool-use results for its size. But evidence quality differs by row. SWE-bench Verified 46.4 and SWE-bench Pro 14.4 are vendor-run; Terminal-Bench 2.1 ~9% is independently sourced from Artificial Analysis; a MiniCPM5-2B Terminal-Bench 4.0 result is not yet published; and the current v4.3 overall score of 14 is explicitly estimated pending a full independent evaluation.

The fairest conclusion is not that a 2B model "beats" every larger model, nor that it suddenly fell from 23 to 14. It is that MiniCPM5-2B has unusually high capability density, while its most impressive coding-agent claims still need matched independent reproduction and the benchmark-version changes must be kept visible.