Terminal-Bench 4.0 Reality Check: 66 Tasks, 5-Trial Harbor Runs vs 3-Trial AA, and Why 2.1 Scores Don’t Transfer

Terminal-Bench 4.0 is a benchmark change, not a model regression chart

Terminal-Bench 4.0 is easy to misread because the name looks like a normal sequel. The maintainers explicitly say otherwise. Version 4.0 recalibrates task resources, fixes task instructions/environments/verifiers, removes saturated or problematic tasks, and changes the agent environment. Under the benchmark's new semantic-versioning policy, those are breaking changes that require rerunning trials.

That is the core rule for interpreting current leaderboards: a Terminal-Bench 2.1 score and a Terminal-Bench 4.0 score are not two points on one clean time series. A model can move sharply between them without proving either a real capability regression or benchmark contamination.

Primary and independent sources:

• Terminal-Bench 4.0 release article
• Terminal-Bench 4 dataset on Harbor Hub
• Terminal-Bench v4.0.0 GitHub release
• Artificial Analysis Terminal-Bench v4.0 evaluation
• Artificial Analysis Intelligence Benchmarking methodology

What actually changed in 4.0

The maintainers describe three important categories of change.

First, resources were recalibrated. Every task now has a flat eight-hour agent timeout, and the team says frontier models now rarely hit timeouts. The resource calibration was designed to reduce infrastructure noise by giving tasks substantial headroom.

Second, the task set changed. The maintainers removed eight tasks: two because of saturation, two because of refusals, two because public solutions existed, and two because of unresolved quality or platform-compatibility problems. They also say 19 tasks were fixed, including changes to instructions, environments or verifiers.

Third, the benchmark is now explicitly continuous and semantically versioned. The maintainers say a major x version change requires rerunning experiments when the agent environment, task set, prompts, tools, data or resource envelope changes. That is exactly what happened from the prior major line to 4.0.

The GitHub v4.0.0 release was published on August 26, 2026 and links the release article, dataset and exact Harbor command for the pinned dataset.

The current dataset is 66 tasks

Harbor Hub currently exposes 66 of 66 tasks for Terminal-Bench 4. The task mix is deliberately broad: software work sits beside scientific computing, data processing, system operations, security, hardware and media-oriented tasks. Some tasks require GPUs or multi-container environments.

This makes Terminal-Bench different from a repository-patching benchmark. It measures an agent operating through a terminal inside a task-specific environment, and each task has its own verifier. A leaderboard score therefore reflects more than the base model: the agent scaffold, tool interface, reasoning configuration, environment, retry policy and resource limits all matter.

Harbor's public leaderboard jobs expose five trials per task

Several official public Terminal-Bench 4.0 jobs on Harbor Hub expose 330 finished trial slots. With 66 tasks in the dataset, that is five attempts per task.

For example, the public Opus 5 / Claude Code job shows 330/330 finished, 324 completed and six errors, an average reward of 0.53, 6.59 billion total tokens and $6,060 total model cost. Harbor reports about $18.36 per finished trial for that run.

The public Sonnet 5 / Claude Code job also shows 330/330 finished, but 293 completed and 37 errors. Harbor reports 21.56 billion total tokens and $9,603.86 total cost, or $29.10 per finished trial. The benchmark maintainers separately note that Sonnet 5 sometimes hit timeouts or output-token limits and used far more tokens than Opus 5.

Those numbers are useful evidence about the cost and failure modes of a concrete leaderboard run. They are not universal API cost or latency estimates. Harbor jobs can run many trials concurrently, so wall-clock duration for the whole job is not the same thing as single-task latency.

Artificial Analysis uses a different evaluation implementation

Artificial Analysis independently evaluates the same 66-task Terminal-Bench 4.0 dataset, but its published methodology is not identical to the official Harbor leaderboard jobs.

Artificial Analysis says it uses mini-SWE-agent v2.4.6, averages pass@1 across three repeats per task, caps the agent at 500 steps, puts a 30-second timeout on each command, allows each task's own timeout up to eight hours, and uses the mini-SWE-agent interactive configuration with native bash. It also says there is no context compaction or summarization: the agent sees its full transcript.

That is a meaningful methodological difference from official Harbor jobs labeled with agent scaffolds such as Claude Code or Codex, and from the public five-attempt pattern visible in those 330-trial jobs.

As of this review, Artificial Analysis's current page reports GPT-6 Astra (xhigh) at 59.6%, GPT-6 Astra (max) at 59.1%, and Claude Fable 5.1 with Adaptive Reasoning, Xhigh Effort and Default Fallback at 55.1%. These are useful independent measurements under Artificial Analysis's specified harness. They should not be silently substituted for a Harbor leaderboard row produced with another agent scaffold or retry count.

The model label matters too. Astra (xhigh), Astra (max), and a Fable configuration with adaptive reasoning and fallback are different runtime configurations, not just interchangeable names for a foundation checkpoint.

Why repeat count matters

Three repeats and five repeats can both estimate pass@1, but they do not produce identical sampling uncertainty. Agentic tasks can have meaningful run-to-run variance because the model may choose different commands, plans or recovery paths.

A score should therefore be archived with at least:

• exact Terminal-Bench version;
• exact task count;
• agent/harness and version;
• model endpoint and reasoning configuration;
• number of repeats;
• environment/provider;
• timeout and step limits;
• tool permissions;
• error-handling policy;
• run date; and
• raw or inspectable trial artifacts when available.

Without those fields, two numbers carrying the label “Terminal-Bench 4.0” can still describe materially different experiments.

Cost and latency are configuration-specific

Terminal-Bench 4.0 does not have one benchmark-wide token budget, price, context window or latency. Those are properties of the evaluated model, agent and run configuration.

The official Harbor jobs demonstrate the scale difference clearly: Opus 5's public run used about 6.59B tokens while Sonnet 5's used about 21.56B. The maintainers specifically call out Sonnet's high token use and output-limit failures. That is evidence that efficiency can diverge substantially even before comparing accuracy.

Artificial Analysis publishes separate token, cost and time-per-task views, but its time metric is tied to its own implementation and provider measurements. It should not be compared directly with Harbor's total concurrent job wall time as if both were the same latency metric.

For production evaluation, teams should measure their own end-to-end completion time, retries, token use, cache behavior and total provider bill with the exact agent stack they plan to deploy.

SWE-bench Verified and SWE-bench Pro are separate

SWE-bench Verified and SWE-bench Pro are not Terminal-Bench 4.0. A Terminal-Bench result should not be relabeled or used to infer either SWE-bench score.

SWE-bench-style evaluations focus on software-repository issue resolution under their own task sets and harnesses. Terminal-Bench covers a broader terminal-agent workload with task-specific environments and verifiers. The benchmark families answer different questions.

For model comparisons, keep the evidence in separate columns: Terminal-Bench 4.0, SWE-bench Verified, SWE-bench Pro, and any other coding or agentic suite should each retain its own version, harness and provenance.

Public reaction: score swings are real; contamination claims are not proven by them

A September 8 r/GeminiAI discussion reacted to a large difference between Gemini 3.8 results on Terminal-Bench 2.1 and 4.0. Commenters proposed explanations ranging from task-distribution and harness changes to benchmark leakage or “benchmaxxing.”

That discussion is useful as evidence of public confusion and suspicion, not as proof of contamination. It is self-selected commentary with no controlled experiment demonstrating that a specific model trained on benchmark answers.

The maintainers provide a directly documented explanation for why scores can move: 4.0 changed resources, fixed tasks, removed tasks and is formally classified as a breaking benchmark revision. Dataset contamination remains a general concern for public benchmarks, but a 2.1-to-4.0 score delta alone cannot establish it.

I did not recover a stable primary X post in this bounded review that added reproducible Terminal-Bench 4.0 evidence, so no X quote or invented consensus is included.

Public discussion:

• r/GeminiAI Terminal-Bench 2.1 vs 4.0 discussion, September 8, 2026

Practical verdict

Terminal-Bench 4.0 is most useful when treated as a versioned agent-system evaluation, not a single timeless model IQ score.

Its 66-task dataset is intentionally broad and resource-heavy. The maintainers changed the task set and environment enough to require reruns. Official Harbor jobs expose five trials per task in several public leaderboard runs, while Artificial Analysis independently uses mini-SWE-agent v2.4.6 and three repeats per task. Those implementation choices can materially affect results.

The clean comparison rule is therefore simple: compare models inside the same frozen benchmark version and evaluation implementation first. When comparing across leaderboards, carry the harness, repeat count, reasoning mode, endpoint, cost and failure policy with the score.

And do not turn a Terminal-Bench 2.1-to-4.0 score change into a capability regression—or a contamination accusation—without a controlled, version-pinned experiment that supports that conclusion.