Gemini 3.8 Flash Reality Check: SWE-Bench Pro, Cyber Variant, Token Costs and Benchmark Drift

Google released Gemini 3.8 Flash on September 2, 2026 as a stable, generally available Flash-tier model aimed at long-horizon software engineering, autonomous agents and complex enterprise workflows. The exact Gemini API model ID is gemini-3.8-flash. Google documents a 1,048,576-token input limit and 65,536-token output limit, with text, image, video, audio and PDF input and text output. Thinking levels are low, medium and high; minimal is not supported.

The important qualification is that 3.8 Flash is not simply “3.7 Flash but smarter at the same cost.” Google explicitly says the new model may take more reasoning steps, make more iterative tool calls and consume more tokens on difficult work. That means the per-token price can stay flat while the cost and latency of a completed workflow rise.

Pricing: introductory rates end on December 31

Google’s paid standard API price through December 31, 2026 is $0.75 per million input tokens and $3.75 per million output tokens, with thinking tokens billed as output. On January 1, 2027, the standard rate is scheduled to become $1.50/M input and $7.50/M output. Current context-cache reads cost $0.075/M tokens, also doubling to $0.15/M on January 1. Batch and Flex are currently half the standard token rate, while Priority inference is priced higher.

This matters for agent evaluation because “price per million tokens” and “cost per successful task” are not the same metric. A model that retries, reasons longer or invokes tools more often can cost more per completed job even if its token tariff is unchanged.

Coding benchmarks: keep the harness attached to the score

Google Cloud’s developer guide reports 61.6% on SWE-Bench Pro for Gemini 3.8 Flash versus 60.4% for Gemini 3.7 Flash. The same guide reports 90.8% versus 81.6% on Terminal-bench 2.1, 51.9% versus 48.0% on SWE-Atlas and 38.1% versus 30.9% on τ³-bench Banking.

Those numbers should not be collapsed into a generic “coding score.” They test different behavior, and Google’s separate evaluation-methodology material uses benchmark-specific harnesses. For example, DeepSWE v1.1 is run with mini-swe-agent for Google’s model, while Terminal-Bench uses the Terminus family of agent harnesses. The current DeepMind evaluation table also shows 89.4% for Terminal-Bench 2.1 rather than the 90.8% in the Google Cloud developer guide. Both are Google surfaces, so the defensible conclusion is that configuration or evaluation snapshot matters; 90.8 and 89.4 should not be treated as one universal immutable number.

SWE-bench Verified is not SWE-Bench Pro

SWE-bench Verified and SWE-Bench Pro remain separate evaluations. In this verification pass, I found a primary Google source for the 61.6% SWE-Bench Pro result, but not a primary Google or benchmark-operator record establishing an exact Gemini 3.8 Flash score on SWE-bench Verified. Secondary sites now circulate an 80.0% Verified figure attributed to another evaluator, but without a directly verified primary run receipt here, it is safer to leave the exact Verified result unresolved than to silently substitute it for Pro.

That distinction is especially important when comparing models: a score on Pro cannot be ranked against a score on Verified as if both came from the same task set, scaffold and tool policy.

Broader agent and multimodal results

Google DeepMind’s current methodology table reports 73.7 on DeepSWE v1.1, 61.4 on Vals Finance Agent v2, 10.0 on Harvey Legal Agent, 54.9 on HLE-Verified, 59.0 on OSWorld 2.0 partial scoring, 86.2 on CharXiv and 87.8 on the agentic version of LVBench. The methodology notes matter: several competitor numbers are provider-reported rather than recomputed by Google, smaller evaluations can be averaged across multiple trials, OSWorld uses a specified task release and step budget, and HLE-Verified uses a revised 1,811-question set rather than the original full HLE.

Google’s model card adds another caution: some performance and safety results use improved evaluations and are not directly comparable with previous Gemini model cards. Versioned benchmark tables are evidence about a particular evaluation setup, not timeless model constants.

Gemini 3.8 Flash Cyber is a restricted sibling, not the public API model

Google launched Gemini 3.8 Flash Cyber alongside the standard model, but access is restricted to trusted defenders through the Fairwind Program. Google says both variants share foundational intelligence, while the Cyber variant is tailored for vulnerability detection and automated patching.

For Gemini 3.8 Flash Cyber, Google reports 86.2% pass@1 on CyberGym. On an internal real-world vulnerability benchmark spanning more than 1,200 historical vulnerabilities across 20 programming languages, Google reports 71.0%, versus 58.9% for 3.7 Flash and 46.6% for 3.5 Flash Cyber. Because that dataset is internal, it cannot currently be reproduced from the public description alone.

CWE-Bench is more externally anchored. Google reports 47.2% pass@1 for 3.8 Flash Cyber, with the run performed by benchmark owner Collinear AI using an Antigravity harness at high thinking. Google compares this with 47.8% for Fable 5 while emphasizing a lower reported cost per rollout. Even here, the methodology warns that CyberGym comparator results can come from different public leaderboards, model cards or proprietary harnesses. The fair claim is that 3.8 Flash Cyber is competitive on the cited cyber evaluations—not that every percentage in the chart is a controlled apples-to-apples head-to-head.

Independent measurement: benchmark drift changes the headline

Artificial Analysis initially published Gemini 3.8 Flash (high) with a score of 59 on its then-current Intelligence Index and estimated about $0.58 per Index task. It also noted that 3.8 Flash used roughly 30% more output tokens per task than 3.7 Flash in that evaluation.

Artificial Analysis then changed its Intelligence Index to v4.2 on September 4, adding more complex and private evaluations and changing the composition of the index. Its current Gemini 3.8 Flash (high) page reports an Index score of 47, about 311 output tokens per second, 10.35 seconds time to first token, a 1M context window and 140M output tokens across the index evaluation.

The drop from the launch-day 59 to the current 47 should therefore not be described as a 12-point model regression. The evaluation yardstick changed. A valid capability-drift claim would require rerunning both the older and newer model snapshots under one frozen benchmark version and harness.

Public developer feedback: useful signals, not consensus

A structured Reddit report posted September 3 ran Gemini 3.7 Flash and 3.8 Flash on the same 98-task MindTrial set with high thinking and the same Python executor. The author reported 87/98 for 3.7 and 86/98 for 3.8, but runtime increased from about 63 to 106 minutes, Python calls from 604 to 741 and reasoning tokens from roughly 660,000 to 1.57 million. That is one self-selected user run on a benchmark whose provenance was not independently audited here, so it is not proof of a general regression. It is, however, directionally consistent with Google’s own warning that 3.8 can “work harder” and spend more tokens.

A separate Google AI Developers Forum report published September 6 describes a repeatable tool-calling regression in which otherwise identical prompts reportedly stop invoking a tool when reflective first-person prose is added. That is a concrete bug report with a reproducible setup, but it is still a single public report until independently reproduced or acknowledged by Google.

Other forum and Reddit threads are mixed: some developers report stronger codebase analysis or successful difficult fixes, while others complain about longer loops, quota consumption or mistakes. These discussions are self-selected and cannot establish population-level satisfaction. I did not find a sufficiently reproducible, attributable technical X post during this bounded review, so no X “consensus” is claimed.

Practical choice: evaluate cost per successful workflow

Gemini 3.8 Flash is a meaningful new stable option for tool-heavy coding and agent workloads, particularly where a 1M-token context window, multimodal input and computer-use support are useful. The strongest evidence for an upgrade over 3.7 is in long-horizon and terminal-style agent evaluations, while the reported SWE-Bench Pro gain is much smaller.

For production selection, test 3.7 and 3.8 on the same repository/task set with the same thinking level, tool permissions, retry budget and stop conditions. Record success rate, wall-clock latency, input/output/thinking tokens, tool-call count, retry rate and total cost per accepted result. Also retest after January 1, 2027 if price is material, because Google’s announced standard token rates double then.

For security work, do not transfer the Cyber variant’s benchmark scores onto the public gemini-3.8-flash endpoint. Cyber is a separately controlled access path with benchmark-specific harnesses and different safety treatment. Likewise, do not rank SWE-bench Verified, SWE-Bench Pro, Terminal-Bench, CyberGym and CWE-Bench as though they measure the same capability.

The evidence today supports a narrower conclusion: Gemini 3.8 Flash is a fast, comparatively inexpensive and increasingly agent-oriented model, but its greater diligence can increase token and tool consumption, benchmark versions can materially change leaderboard headlines, and exact harness identity is essential before making a “best coding model” or “best cyber model” claim.