Gemini 3.8 Flash Reality Check: 80.0 Vals Verified, 61.6 Pro, and Google's Terminal-Bench Split
Gemini 3.8 Flash has an independent 80.0% Vals SWE-bench Verified result and a Google-reported 61.6% Pro result, while Google's own pages currently disagree on Terminal-Bench 2.1.
What Gemini 3.8 Flash actually is
Google released Gemini 3.8 Flash on September 2, 2026 as the next iteration of Gemini 3.7 Flash. The public API model ID is gemini-3.8-flash. Google documents a 1,048,576-token input limit, 65,536-token maximum output, text/image/video/audio/PDF input, text output, and LOW, MEDIUM, and HIGH thinking levels.
The launch also introduced Gemini 3.8 Flash Cyber, but it is a distinct restricted deployment for trusted defenders through Google's Fairwind Program. CyberGym, CWE-Bench, vulnerability-discovery and patching claims for that restricted variant should not be silently transferred to the generally available gemini-3.8-flash model.
Primary sources:
SWE-bench Verified: Vals lists 80.0%, with a timestamp caveat
The strongest independent software-engineering evidence I found is Vals AI's SWE-bench Verified evaluation. Vals uses the human-validated 500-task Verified split and a minimal mini-swe-agent, bash-only harness. Every model gets the same single bash tool and must navigate, search, edit and produce a patch with ordinary command-line tools. Vals says provider defaults are used except that the maximum token limit is set as high as possible.
The current Vals model record identifies Gemini 3.8 Flash at high reasoning, and its SWE-bench record carries a ±1.79 percentage-point uncertainty indicator. The public leaderboard data mirrored from the Vals result records the overall Gemini 3.8 Flash result as 80.0%. The direct Vals chart also shows the model's difficulty-band resolution rates as approximately 90% for <15-minute tasks, 78% for 15-minute-to-1-hour tasks, 52% for 1–4-hour tasks, and 33% for >4-hour tasks.
There is an important metadata problem: Vals currently labels the SWE-bench Verified page "Updated 9/1/2026", while Google says Gemini 3.8 Flash was released on September 2. The page also says the benchmark is archived because performance has saturated and that Vals no longer runs it on new releases, yet Gemini 3.8 Flash appears in the preserved results. This could reflect pre-release access, stale page metadata or a later result added without changing the displayed update date. I found no authoritative explanation.
So the fair label is: independent, harness-documented 80.0% result, but with an unresolved public timestamp inconsistency. I do not claim the run itself happened on September 1.
Sources:
SWE-bench Pro: Google reports 61.6%, but it is not the Vals Verified run
Google Cloud's current developer guide reports 61.6% on SWE-Bench Pro for Gemini 3.8 Flash versus 60.4% for Gemini 3.7 Flash. That is a first-party Google result.
This score must stay separate from the Vals 80.0% Verified result. Verified and Pro are different benchmark suites with different task construction and evaluation histories. Google's developer guide does not, on the page where it reports 61.6%, pin the exact SWE-bench Pro task revision, agent scaffold, task count, retry policy or raw trajectories. Google's separate four-page DeepMind evaluation-methodology document currently does not include SWE-bench Pro in its displayed results table.
I therefore found no basis to subtract 61.6 from 80.0, combine them, or claim the lower Pro percentage means regression. The defensible description is 61.6% vendor-reported SWE-bench Pro, not independently reproduced here on a matched harness.
Source:
Google's own Terminal-Bench 2.1 documentation currently disagrees
A more immediate reproducibility warning appears inside Google's own documentation.
The Google Cloud developer guide currently reports 90.8% Terminal-bench 2.1 for Gemini 3.8 Flash and 81.6% for 3.7 Flash. But Google DeepMind's current evaluation-methodology PDF shows 89.4% for Gemini 3.8 Flash and 85.8% for 3.7 Flash.
The DeepMind methodology document is more explicit about its setup: Gemini Terminal-Bench 2.1 results are self-computed, use the default Terminus 2 agent harness, and Gemini results use the gemini-3.8-flash API model ID with default sampling unless otherwise noted. The PDF also says all Gemini scores are pass@1 unless specified otherwise.
The two official pages therefore disagree by 1.4 points for 3.8 Flash and 4.2 points for 3.7 Flash. I did not find a Google note explaining whether the Cloud guide uses a different snapshot, harness revision, task revision or later rerun. Until Google reconciles those values, it is safer to cite the source and setup instead of presenting one Terminal-Bench number as the uncontested score.
Sources:
DeepSWE and HLE-Verified are useful, but still Google-run for this model
Google DeepMind's methodology currently reports 73.7% on DeepSWE v1.1 for Gemini 3.8 Flash. It says the Gemini result is self-computed with mini-swe-agent and high thinking, while comparison rows are taken from the public DeepSWE leaderboard at each model's highest listed thinking level.
For HLE-Verified, Google reports 54.9%. Its methodology says this is self-computed on the full 1,811-item verified set, comprising 668 verified original HLE items plus 1,143 revised items, while excluding 689 original items identified as uncertain.
These results are informative, but they are different evaluation families and should not be used to validate the SWE-bench scores.
Independent API measurements show why per-token price is not the whole bill
Google's introductory Gemini 3.8 Flash API price is $0.75 per million input tokens and $3.75 per million output tokens through December 31, 2026. Google says the regular price from January 1, 2027 will be $1.50/M input and $7.50/M output.
Google also explicitly warns that 3.8 Flash improves accuracy at the cost of higher token consumption than 3.7 Flash, and recommends lower thinking levels or 3.7 Flash when compute efficiency matters.
Artificial Analysis provides a separate, point-in-time independent view of the high configuration. Its current page reports:
- 41 on Intelligence Index v4.3;
- about 277.3 output tokens/second on Google's API;
- approximately 15.95 seconds time to first token;
- $1.24 per Intelligence Index task;
- about 170 million output tokens consumed across the Index evaluation;
- a 1M-token context window.
Those figures are not SWE-bench measurements and they are not service-level guarantees. They do, however, support the practical point that a low token price can coexist with meaningful end-to-end cost and latency when a model reasons longer or emits more tokens.
Source:
Public feedback: launch excitement is not independent validation
Google DeepMind's September 2 X launch post presents 3.8 Flash as its most intelligent model with gains in software engineering, agentic tasks and multi-step reasoning. That is useful primary evidence for what Google announced, but it is vendor communication, not a controlled independent rerun.
A more useful practitioner datapoint comes from developer Elzo Brito's September 2 hands-on report. He describes Gemini 3.8 Flash successfully staying on track through a long Antigravity workflow with project memory, tools and a large context. He also says pieces of the produced lesson were incomplete and explicitly frames his conclusion as an opinion from one operational workflow, not a universal benchmark or head-to-head.
That anecdote is consistent with Google's long-horizon positioning, but it does not reproduce Google's DeepSWE, Terminal-Bench or SWE-bench Pro results. I found no controlled same-harness X rerun that resolves the official Terminal-Bench discrepancy or independently reproduces the 61.6 Pro score.
Source:
Practical verdict
Gemini 3.8 Flash has a stronger evidence trail than a launch-table-only model: Vals exposes a consistent bash-only SWE-bench Verified harness and lists an 80.0% result, while Artificial Analysis supplies independent API speed, cost and composite-evaluation measurements.
But the evidence also has unusually clear bookkeeping warnings. Vals' displayed archive/update date predates Google's public launch, Google's two official pages currently disagree on Terminal-Bench 2.1, and the 61.6 SWE-bench Pro row is not accompanied by enough public harness detail to reproduce it exactly.
The most defensible reading today is therefore not "3.8 Flash wins coding." It is: a generally available 1M-context Flash model with credible independent Verified evidence, strong first-party long-horizon results, competitive introductory pricing, and unresolved evaluation-version metadata that matters for fair ranking.
The next evidence that would materially improve confidence is a dated, version-pinned independent SWE-bench Pro rerun, a Google clarification of the 89.4-versus-90.8 Terminal-Bench difference, and a Vals clarification of when its Gemini 3.8 Flash Verified run was executed.
This article is built from the source material below. Open the originals for full context and the latest updates.
