AIM Enterprise Reality Check: 69 Tasks Put Opus 5, GPT-6 Astra, Fable 5.1 and Grok 4.6 Within 2.9 Points—but Judges Disagree 32.7% of the Time
AIM Enterprise tests 16 models across 69 business tasks. Its top four are within 2.9 points, while two LLM judges disagree sharply on 32.7% of individual scores. We audit harness, cost and benchmark limits.
What changed on September 8
AIMultiple updated AIM Enterprise on September 8, 2026, positioning it as a third-party agentic benchmark for routine enterprise work rather than a coding-only or academic reasoning leaderboard. The current field covers 69 tasks across strategy, marketing, HR, sales, operations, IT, finance and smaller business-process categories. Tasks ask an agent to research the live web and produce a fixed results.csv, usually with 10 rows, eight columns and explicit rules for integer fields.
Primary benchmark:
The most defensible headline is not that one frontier model has decisively won. AIMultiple says Claude Opus 5, GPT-6 Astra, Claude Fable 5.1 and Grok 4.6 are separated by only 2.9 points. Opus 5 scored higher than Astra on 39 of 69 tasks while Astra scored higher on 30; Astra led the field on 23 individual tasks, Opus 5 on 22 and Fable 5.1 on 16.
That is useful evidence for enterprise-agent buyers, but it is not a clean model-only comparison.
Cost results are interesting, but they are reconstructed—not actual invoices
AIMultiple reports an average benchmark cost of $1.76 per task for GPT-6 Astra, $1.73 for Claude Opus 5, $3.19 for Claude Fable 5.1, $1.09 for Grok 4.6, $0.167 for GPT-5.6 Sol, and $0.016 for GPT-5.6 Luna. It also reports a 0.68 correlation between cost per task and benchmark score, versus only 0.04 between median task time and score.
Those figures should not be read as the actual bills paid for the runs. AIMultiple says the runs used subscriptions, then it recomputed costs from recorded token use using frozen list-rate snapshots: mostly OpenRouter prices from August 27, with Astra and Gemini 3.8 Flash rates read September 5 and Anthropic's September 5 Fable 5.1 pricing because OpenRouter did not list Fable 5.1. Claude Code's cache, web-search and Haiku subagent costs were also included.
For current API context, OpenAI lists gpt-6-astra at $10/M input and $50/M output tokens, with a 1,050,000-token context window and 128,000 maximum output tokens. Anthropic lists claude-fable-5-1 at $10/M input, $50/M output and $0.25/M cache reads, with 1M context and 128K maximum output. xAI lists grok-4.6 at $2/M input and $6/M output below its higher-context pricing threshold, with a 500K context window.
Official model references:
Current raw token prices still do not tell you which model is cheapest per successful enterprise workflow. Tool calls, reasoning length, cache behavior, agent harness, retry rate and output quality all affect end-to-end cost.
The harness is part of the result
The benchmark used 16 models in 17 setups. Eight ran through opencode 1.15.13 via OpenRouter; the rest used vendor agent programs including Claude Code, Codex and Grok Build. Fable 5.1 used Claude Code 2.1.258, while Astra used Codex 0.153.4.
Every setup received the same frozen prompt, live-web access through a scraping API and a one-hour limit. But the reasoning settings were not normalized. AIMultiple let each agent program use its default: Claude Code ran the Claude models at high effort, Codex ran GPT-6 Astra at medium, Grok Build ran Grok 4.6 at high, and opencode left effort to the provider default.
That matters. A benchmark of model + agent program + default effort + tools is highly relevant to practical deployment, but it should not be interpreted as a controlled comparison of base model capability alone.
A small within-model check reinforces the point: Claude Sonnet 5 ran under both Claude Code and opencode and scored 47.9 versus 47.5. One pair is far too little to estimate a general harness effect, but it shows that the benchmark itself cannot cleanly separate model quality from orchestration.
Failures count, which is good—but repeatability is still limited
Across 1,173 setup-task pairs, the runs produced 1,140 files. Missing files normally scored zero. A retry was allowed only for the one-hour limit or a provider error. The benchmark reports five stalls, 18 provider errors and eight Qwen 3.8 Max runs stopped by opencode's loop guard.
Before LLM judging, deterministic checks rejected malformed outputs for issues such as wrong columns, row counts, CSV parsing, empty cells, duplicates or required sort order. Of the produced files, 1,130 passed and went to the two LLM judges.
This is stronger than silently discarding failures, because operational reliability becomes part of the score. But each completed file was scored once, and the same setup-task pair was not repeatedly rerun to estimate ordinary stochastic variance.
The biggest warning is the judge disagreement
AIM Enterprise uses two judge models: GPT-5.6 Sol via Codex CLI and Claude Opus 5 via Claude Code, both at high reasoning effort with live web access. Each column is judged separately against anonymized, shuffled answers, and the two rank orders are combined.
AIMultiple reports that on 32.7% of individual scores the judges differed by more than one quarter of the scoring scale, and those disagreements were not manually reviewed. Twelve of the 17 setups also changed rank depending on which judge was used.
A separate visible-name diagnostic found directional family effects: the GPT judge scored GPT answers higher relative to the Opus judge, while the Opus judge scored Anthropic answers higher relative to the GPT judge. In the anonymized production run, the GPT judge ranked Astra first while the Opus judge ranked Opus 5 first.
This does not invalidate the benchmark, but it means small gaps—especially the 2.9-point spread across the top four—should not be treated as precise capability differences without human adjudication or additional judge families.
The 4,000 resamples measure task selection, not rerun variance
AIMultiple resampled the 69-task set 4,000 times to see how much rankings change when the task mix changes. That is useful for measuring sensitivity to task selection. It does not estimate run-to-run variance, because the same task was not executed repeatedly by the same setup.
The publisher also states that the numeric scale is relative to the field of competitors: dropping a setup changes everyone else's score. Therefore the absolute numbers should not be compared to a prior AIM Enterprise run with a different model field, and should not be compared numerically with unrelated leaderboards.
This is an important benchmark-hygiene point. A model scoring 60 here is not "twice as capable" as one scoring 30, and a 60 here cannot be compared directly with 60 on another benchmark.
SWE-bench Verified and SWE-bench Pro are separate evidence
AIM Enterprise is not SWE-bench Verified, SWE-bench Pro, Terminal-Bench or another software-engineering benchmark. Its tasks focus on enterprise research and structured business outputs. None of its scores should be transferred into coding-agent rankings.
For GPT-6 Astra, Claude Fable 5.1, Claude Opus 5 or Grok 4.6, SWE-bench Verified and SWE-bench Pro must remain separate evidence cells with their own revision, harness, task count, retries, date and provenance. This article does not infer a SWE-bench Verified or SWE-bench Pro score from AIM Enterprise, nor does it merge the two variants.
OpenAI's Astra system-card material also references SWE-Bench Verified in a safety/control evaluation context. That is not the same thing as a standard end-to-end SWE-bench software-engineering result and should not be silently repackaged as one.
What the benchmark does tell us
Three conclusions are reasonably supported.
First, the current frontier is bunched rather than separated on this task set. The top four are within 2.9 points, while Fable 5.1 and Astra show substantial gains over their named predecessors in the publisher's same-run comparison.
Second, cost efficiency varies dramatically. A much cheaper model can be materially behind the leaders yet still be the rational choice for high-volume workflows where a human reviews the output.
Third, evaluation design dominates close calls. Default reasoning effort, agent harness, provider reliability, judge family, task selection and relative scoring can all move a ranking. A procurement decision based only on the overall score would throw away the most useful information in the benchmark.
Public feedback is not a second benchmark
Fresh exact-match searches did not surface a reproducible X or Reddit discussion specifically evaluating AIM Enterprise, and the benchmark page itself did not provide an independent human-adjudicated review of the high-disagreement rows. There is therefore not enough sourceable public evidence to claim community consensus about the ranking.
That absence is worth stating directly. General enthusiasm or complaints about Astra, Fable, Opus or Grok are not evidence about this 69-task setup unless the commenter actually reran or audited the benchmark.
Practical verdict
AIM Enterprise is a valuable new third-party enterprise-agent benchmark, particularly because it publishes more methodology than a typical launch chart: task mix, agent programs, effort defaults, failure handling, deterministic checks, judging method and cost reconstruction are all described.
Its strongest result is also its strongest caution. Opus 5, GPT-6 Astra, Fable 5.1 and Grok 4.6 are close enough that the 32.7% judge-disagreement rate and heterogeneous harnesses matter more than a one- or two-point leaderboard gap.
For a serious deployment, reproduce a representative subset with the same harness and reasoning budget, run each task multiple times, add human review for disputed outputs, measure total successful-task cost rather than raw token price, and keep coding benchmarks such as SWE-bench Verified and SWE-bench Pro in their own clearly labeled comparison table.
This article is built from the source material below. Open the originals for full context and the latest updates.
