GPT-6 Astra ARC-AGI-3 Reality Check: 99.9% vs 62.7% Depends on the Harness

The 99.9% headline needs a harness label

OpenAI introduced GPT-6 Astra on September 3, 2026, with a headline 99.9% ARC-AGI-3 result. That number is real in ARC Prize's published evaluation, but it is not the only Astra result on the same benchmark.

ARC Prize independently reports two materially different results on the 55-environment ARC-AGI-3 Semi-Private set:

• 62.7% with ARC Prize's Standard harness at max reasoning, at a reported evaluation cost of $26,098.
• 99.9% with ARC Prize's Provider Adapter at high reasoning, at a reported evaluation cost of $18,817.

Those scores should be written with their harnesses attached. Saying only "Astra scores 99.9% on ARC-AGI-3" removes an experimental variable that changes the measured result by 37.2 percentage points.

What the two harnesses actually do

ARC Prize describes its Standard harness as deliberately minimal and provider-neutral. It exposes the game interface to the model through a common interaction loop and keeps visible notes/state in a way that is intended to make providers easier to compare.

The Provider Adapter instead allows provider-native context-management behavior. For GPT-6 Astra, ARC Prize says this includes preservation of opaque reasoning state and provider compaction across the interaction. That makes the adapter closer to how a provider may want its long-horizon model used, but it also means part of the state-management system is no longer transparent or identical across providers.

ARC Prize therefore now reports the two harness families separately. That is the right methodological choice: the model plus harness is the evaluated system when harness behavior materially affects performance.

The adapter was faster and used fewer tokens on commonly solved cases

ARC Prize gives a useful paired comparison rather than only the endpoint scores. Across 167 game-reasoning pairs that both harnesses solved, it reports the Provider Adapter at approximately 3.66× faster aggregate elapsed time and using 49% fewer total tokens.

This is not a claim that every Provider Adapter run is 3.66× faster. The comparison is restricted to pairs solved by both harnesses, so it conditions on successful overlap. Still, it is strong evidence that context handling can materially reduce repeated work as well as increase completion.

The cost table also shows an unusual pattern. Under the Standard harness, higher reasoning did not simply mean higher total cost:

• max: 62.7%, $26,098
• xhigh: 59.3%, $37,317
• high: 54.8%, $40,705
• medium: 38.6%, $48,090
• low: 17.5%, $38,166
• none: 35.2%, $49,791

ARC Prize explains that stronger reasoning can reduce the number of actions required to solve games, which reduces the number of model calls and tokens. In other words, more expensive reasoning per step can still be cheaper for the full task if it avoids enough wasted steps.

Under the Provider Adapter harness, scores were tightly clustered near saturation: 98.6% at max, 98.4% at xhigh, 99.9% at high, 98.4% at medium, 98.0% at low and 96.7% with no reasoning effort in the ARC test configuration. Those values should not be collapsed into a single “Astra score”; they are separate configuration results.

ARC Prize's “none” row is an evaluation configuration. OpenAI's public API documentation for gpt-6-astra lists low, medium, high, xhigh and max reasoning levels, not none, so the ARC row should not be mistaken for a documented production API setting.

ARC-AGI-3 is 135 environments, but Astra's headline is the 55-environment Semi-Private set

The ARC-AGI-3 technical report defines three environment groups:

• 25 Public Demo environments
• 55 Semi-Private environments
• 55 Fully Private environments

The Semi-Private set is the external-API evaluation set used for published third-party model testing. ARC Prize notes that it carries a small leakage risk because it is exposed through evaluation access. The Fully Private set remains separated from that external API path.

That distinction matters when comparing future scores. A score on Public Demo, Semi-Private or Fully Private should never be moved into another column just because all three belong to ARC-AGI-3.

ARC-AGI-3 also evaluates interactive behavior rather than one-shot answers. Its Relative Human Action Efficiency methodology compares the number of actions an agent needs with human action counts. The benchmark therefore tests a mixture of perception, exploration, planning, memory and interaction efficiency inside game-like environments.

Human calibration shows solvability, not a direct human-vs-model intelligence ratio

The technical report says ARC Prize tested 10 people per environment during human calibration. An environment was included only when at least two independent first-run participants fully solved it, and ARC Prize reports that every included environment was human-solvable under that calibration procedure.

That is valuable validation that the benchmark is not made of impossible tasks. It does not make 99.9% a direct statement that Astra has 99.9% of human general intelligence, nor does the model's dollar cost map cleanly onto the compensation paid to human participants. The humans and model are operating under different interfaces, economics and evaluation protocols.

Saturation is not proof of AGI

ARC Prize explicitly warns against interpreting saturation as proof of artificial general intelligence. ARC-AGI-3 consists of bounded, deterministic environments with a defined action space and scoring system. A model can become extremely strong at that evaluation family without thereby proving open-ended competence across science, software engineering, social reasoning, physical tasks or real-world uncertainty.

The important result is narrower and still significant: Astra plus the Provider Adapter can solve nearly all of ARC Prize's current Semi-Private ARC-AGI-3 environment set under the published evaluation protocol. The Standard harness result shows that the surrounding context-management system is a major part of that capability.

The model itself is expensive, and long-context requests can cost more than the headline rate

OpenAI's API documentation identifies the public model as gpt-6-astra with a 1,050,000-token context window, 128,000 maximum output tokens and an April 30, 2026 knowledge cutoff.

Standard API list pricing is $10 per million input tokens, $1 per million cached-input tokens, $12.50 per million cache-write tokens and $50 per million output tokens. OpenAI also states that requests with more than 272K input tokens are priced at 2× input and cache rates and 1.5× output for the full request.

Those rates help explain why a full interactive benchmark can cost tens of thousands of dollars. ARC Prize's $26,098 and $18,817 figures are reported evaluation totals for particular harness/reasoning configurations; they should not be treated as a universal “cost to use Astra.”

For production work, the useful metric is normally cost per successful task at the required reliability level, not token list price by itself. A context strategy that reduces retries can be cheaper even if it uses a more capable model or higher reasoning setting.

OpenAI's coding scores are separate from ARC-AGI-3

OpenAI's launch evaluation reports several coding and computer-use results for Astra, including 57.9 on Terminal-Bench 4.0, 74.1 on DeepSWE v1.1, 64.5 on FrontierCode 1.1 Extended and 53.3 on FrontierCode 1.1 Main. It also reports 72.6 on OSWorld in roughly 40 minutes in its evaluation setup.

These benchmarks measure different things and use different harnesses. A strong ARC-AGI-3 score should not be converted into a coding score, and a Terminal-Bench or DeepSWE result should not be treated as an ARC result.

SWE-bench Verified and SWE-bench Pro remain separate

OpenAI's current Astra launch page does not provide an exact new SWE-bench Verified or SWE-bench Pro result with a pinned split and evaluation recipe that can be attached to this article. The launch materials instead emphasize Terminal-Bench, DeepSWE and FrontierCode.

So the correct entries here are unknown from the reviewed primary launch evidence, not a number borrowed from an older GPT model, a secondary leaderboard or another coding benchmark. SWE-bench Verified and SWE-bench Pro should also be kept as separate benchmarks rather than treated as interchangeable versions of one score.

Early developer feedback is useful for hypotheses, not consensus

Launch-week public discussion is mixed. In the OpenAI Developer Community, one developer testing Astra on game-modding and reverse-engineering work described it as clearly more capable than the previous Sol generation in many local coding tasks but still prone to patch-and-test loops and weaker global architecture judgment. Another developer reported a claimed false-positive cybersecurity block while performing a reliability audit they described as non-security work.

These are attributable first-hand reports, but they are still self-selected anecdotes. They do not measure failure rates, benchmark accuracy or population-wide user sentiment.

A bounded search did not surface a sufficiently reproducible current X run that pinned the exact Astra model, ARC-AGI-3 harness, reasoning level, cost and replay evidence. No X consensus is inferred.

Why the 62.7-to-99.9 gap is more useful than hiding it

It may be tempting to treat the 62.7% Standard result as an obsolete configuration and advertise only the 99.9% Provider Adapter result. That would throw away one of the most useful findings in the evaluation.

The 37.2-point harness gap demonstrates that long-horizon agent benchmarks can measure more than foundation-model weights. Context preservation, reasoning-state continuity, compaction, retries, action formatting and scaffold design can all alter measured performance and cost.

For researchers and engineering teams, benchmark records should therefore include at least the exact model ID, benchmark split, harness/scaffold version, reasoning effort, context-management policy, action limits, date, cost, token use and latency definition. If a provider-native adapter has access to opaque state unavailable to a provider-neutral harness, that should be visible beside the score.

The fairest summary of the current evidence is not simply “GPT-6 Astra gets 99.9% ARC-AGI-3.” It is: ARC Prize reports 62.7% with its Standard harness and 99.9% with a Provider Adapter on the 55-environment Semi-Private set, with the adapter also showing lower reported cost and substantially better paired token/time efficiency. That result is impressive, but its methodology is part of the result.