E-Commerce Bench Reality Check: GPT-5.6 Sol Earned 14.3×, Fable5 Was More Efficient, and API Cost Is Unscored

What E-Commerce Bench actually measures

E-Commerce Bench is a long-horizon agent benchmark built by researchers from the Qwen Team, Taobao & Tmall Group and collaborators. The paper was submitted to arXiv on August 31, 2026, Qwen publicly announced the benchmark on X on September 3, and Alibaba Cloud published a detailed explainer on September 7.

Instead of asking an agent to finish a short task, the benchmark gives it ¥100,000 and a simulated 2026 calendar. The agent can operate as many as four online stores for 365 simulated days, choosing products, negotiating with suppliers, buying inventory, pricing and listing goods, fulfilling orders, handling returns and managing cash flow. The public environment contains 6,886 products across 60 categories and 576 suppliers, including 152 fraudulent suppliers, plus eight promotions and ten fixed market events.

The primary score is year-end total assets, but the benchmark deliberately reports six additional capability dimensions: negotiation, fraud avoidance, solvency, operational efficiency, execution and learning over time. That matters because the published results do not support a simple “best model at business” ranking.

The headline: GPT-5.6 Sol made the most money

The benchmark evaluates 18 models, each across five independent episodes, for 90 episodes in total. Every model begins with the same ¥100,000 stake.

The strongest mean year-end asset result is GPT-5.6 Sol (max) at ¥1,431,425, roughly 14.3× the opening stake. The strongest open-weight entry is Qwen3.8-Max-Preview at ¥416,252, roughly 4.2× the stake. Ten of the 90 total episodes ended in bankruptcy.

Those numbers are meaningful inside this simulated economy, but five episodes per model is still a small sample for a year-long stochastic-looking agent workflow. The environment is engineered to remove much of the economic randomness, and the leaderboard reports standard deviations, but readers should still resist treating a five-run mean as a precise universal ordering.

The winner on assets is not the winner on safety or efficiency

The most useful result is the capability split.

GPT-5.6 Sol leads on total assets but performs poorly on fraud avoidance. Alibaba's published analysis says it ranks 16th of 18 on that dimension. In the official leaderboard, 18.48% of its procurement spend goes to fraudulent suppliers.

By contrast, Claude Opus 4.7 sends only 0.12% of procurement spend to fraudulent suppliers and records the strongest negotiation score in the published table. Among fraudulent suppliers contacted, the Alibaba analysis reports that only 4.0% ultimately received an order from Opus 4.7, compared with 31.7% for GPT-5.6 Sol.

Operational efficiency tells another story. The benchmark reports Fable5 at ¥479 of profit per tool call, versus ¥363 for GPT-5.6 Sol, while Fable5 uses 59.9% fewer tool calls in the project analysis. This does not mean Fable5 is cheaper to run in dollars; it means it extracted more simulated business profit per environment action.

That distinction is important because “profit per tool call” is an in-world efficiency metric, not an API-cost metric.

Exact identity check: this is Fable 5, not Fable 5.1

The public benchmark repository's shipped model configuration names the evaluated Anthropic entry as claude-fable-5 with maximum reasoning effort. The leaderboard itself labels it Fable5 (max).

That is not evidence for a model called Fable 5.1. A later or differently named Fable release cannot silently inherit this benchmark result. Any Fable 5.1 comparison needs a new run with its own exact model ID and configuration.

The same rule applies to GPT-6 Astra. Astra is not one of the 18 models in the current shipped E-Commerce Bench model registry, so this benchmark provides no Astra asset score, fraud score, efficiency score or overall rank. It would be methodologically wrong to infer one from GPT-5.6 Sol or from Astra's results on unrelated benchmarks.

The harness controls more than the model name

E-Commerce Bench tries to make model comparisons cleaner by fixing the environment around them. The customer-side demand model is deterministic, and supplier prices, concessions, acceptance and walk-away behavior come from a deterministic negotiation kernel. A separate LLM only renders those already-decided supplier actions into natural dialogue.

All evaluated agents receive the same 18 e-commerce tools. The project page says transcript eviction does not begin until the conversation exceeds 120,000 tokens, after which the harness manages context. This means the evaluation is testing performance under a shared long-horizon context-management system rather than simply allowing every vendor's largest native context window to dominate the comparison.

The public configuration also shows an important non-equivalence: the models were not all run at one identical reasoning-effort setting. GPT-5.6 Sol is configured at max, GPT-5.5 at xhigh, Fable5 and several Claude models at max, and Gemini entries at high, while some OpenAI-compatible models do not expose the same cross-vendor effort control. The benchmark standardizes the environment and tools, but it is not an iso-compute or iso-reasoning-budget study.

Long-horizon learning is still weak

The benchmark's AnchorRatio metric asks whether an agent gets better prices when re-ordering the same product from the same supplier. A value of 1.0 means its price ordering is no better than a reshuffling of its own negotiated prices; lower is better.

Across 8,647 repeat purchases, only two of the 18 models fall below 1.0. The median is 1.369. The project reports Qwen3.8-Max-Preview at 0.834 and describes it as the only model showing a clear positive sign of long-horizon price learning under its significance test.

This is arguably more revealing than the asset leaderboard. Many agents can accumulate profit in a fixed economic environment without demonstrating that they reliably preserve and exploit useful experience over hundreds of simulated days.

Pricing and latency: the leaderboard does not answer the deployment-cost question

E-Commerce Bench publishes tool-call counts and simulated profit per tool call, but I did not find a standardized table reporting actual API inference spend, input/output token bills, time-to-first-token, output speed or full wall-clock runtime for every evaluated model.

That omission matters. A model that makes fewer environment calls can still be more expensive if each call uses substantially more tokens or more costly reasoning. Conversely, a faster or cheaper model can be operationally attractive even if it earns less simulated money.

So the published ¥479 versus ¥363 per tool call comparison should not be translated into “Fable5 is 32% cheaper” or any other API-cost claim. It is an environment-action efficiency result only. A deployment-grade follow-up should publish token consumption, vendor price snapshot, cache behavior, retry counts and end-to-end elapsed time alongside the business metrics.

Reproducibility is better than many agent benchmarks, but independent reruns still matter

The benchmark is unusually transparent for a frontier-model agent comparison. The authors publish the paper, project site, code/data repository and model-provider configuration. The repository exposes exact model IDs and provider routes for rerunning the environment, while the economic kernel is designed to keep supplier and customer outcomes reproducible.

There is still a difference between an open benchmark and an independently reproduced leaderboard. The repository notes that the paper's model runs were reached through an internal gateway, while its current public configuration maps the same model names to their vendor endpoints. In this bounded September 7 check, I did not find a separate research group publishing a complete, version-pinned 90-episode rerun with raw trajectories and matching statistics for the headline models.

The right confidence label is therefore: project-run results on an open and rerunnable benchmark, with independent reproduction still limited.

SWE-bench Verified and SWE-bench Pro are separate

E-Commerce Bench is not a software-engineering issue-resolution test. Its paper discusses coding-agent benchmarks in related work, but the E-Commerce Bench leaderboard itself does not produce a SWE-bench Verified score or a SWE-bench Pro score.

Those two SWE-bench families must also remain separate from each other. No E-Commerce Bench asset multiplier, fraud rate or tool-efficiency number can fill a missing SWE-bench Verified or SWE-bench Pro result, and a strong SWE-bench result cannot be used to infer year-long business-operation performance.

For GPT-5.6 Sol, Fable5, Qwen3.8-Max-Preview or any future Astra run, the coding benchmarks and this business-agent benchmark should be reported as distinct evidence.

Public discussion: interest is real, controlled reproduction is not yet visible

Qwen's September 3, 2026 X post publicly introduced the benchmark and highlighted the ¥100,000 starting stake, 365-day horizon, 6,886 products, 576 suppliers and seven-axis evaluation. Because the post comes from the benchmark's own Qwen account, it is vendor/project communication, not independent validation.

A September 5 Reddit post in r/AIDeveloperNews praised the open-source setup and summarized the deterministic negotiation design. That post is useful as evidence of public interest, but it does not provide a pinned independent rerun and should be treated as self-selected commentary rather than measured consensus.

A bounded X search did not surface a stable, attributable independent post containing a controlled same-version rerun with raw results. No claim of broad X consensus is therefore justified.

Practical takeaway

E-Commerce Bench is valuable because it exposes a failure mode that single-number leaderboards often hide: the model that maximizes the headline outcome can still be weak at fraud avoidance, while a model that is more action-efficient can earn less overall.

For teams evaluating autonomous business agents, the most defensible reading is a profile, not a crown. Look at total outcome, safety losses, cash discipline, negotiation, learning, tool-call volume, inference cost, latency and the exact harness together.

The next evidence worth watching is an independent rerun that pins model versions and reasoning settings, reports token/API cost and latency, and adds newer systems such as GPT-6 Astra or an exact Fable 5.1 identity without substituting results from earlier models.