UK AI Security Institute Releases optstop to Cut Waste in LLM Evaluations

AISI releases an adaptive stopping tool for expensive model evaluations

The UK AI Security Institute (AISI) published optstop on August 27, 2026, an open-source Python package designed to reduce unnecessary compute in repeated LLM evaluations. The tool integrates with AISI's Inspect evaluation framework and changes a common assumption in benchmark execution: instead of fixing the same number of trials for every task before an evaluation starts, optstop can keep sampling where uncertainty remains high and stop where estimates have become sufficiently precise or stable.

This is a released open-source evaluation tool and accompanying research method, not a new foundation model or a claim that every AI benchmark can be shortened by the same amount.

Why fixed evaluation budgets can waste compute

Frontier-model evaluations increasingly repeat tasks several times because model outputs are stochastic. A fixed trial count is simple, but uncertainty is rarely distributed evenly. Some model-task combinations become statistically stable quickly, while difficult or noisy cases need more observations. AISI argues that continuing to spend trials on already-stable cases can consume budget that would be more useful elsewhere.

AISI also notes that some frontier evaluations now require hundreds of millions of tokens. As test suites get longer and models become more expensive to run, evaluation efficiency becomes an infrastructure issue as well as a statistical one.

How optstop decides when to stop

optstop treats evaluation as sequential measurement. It monitors uncertainty at both the repeated-task level and the broader grouping level. At the grouping level, the method uses a hierarchical Bayesian model to combine information across tasks while allowing individual task performance to differ.

Sampling can stop under two principal rules: precision, when the credible interval is narrow enough, or stabilisation, when that interval has stopped materially changing. If neither condition is satisfied, the evaluation continues to the full planned budget.

The package also includes a conservatism mechanism for rare-success settings. AISI highlights this because stopping too early is especially risky when a model succeeds only occasionally and those rare successes are exactly what an evaluator is trying to detect. Each stopping decision is recorded so evaluators can inspect why a run ended.

AISI reports large savings in its test conditions

AISI tested optstop across binary, ordinal and continuous scoring settings, using public benchmarks including MATH, GPQA Diamond and WritingBench and spanning low, medium and high expected model performance. It reports that, across the conditions it tested, early stopping saved 57% to 97% of planned trials without changing the score estimates.

Those numbers should be interpreted narrowly. They are results from AISI's specified experimental conditions, not a universal guarantee for arbitrary benchmarks. AISI explicitly notes that leaner evaluation designs with fewer planned trials may save less.

A staged adoption path reduces operational risk

The release supports three practical adoption modes. Evaluators can first run optstop post-hoc on completed data to compare stopped estimates against full-run estimates; next use shadow mode, where the evaluation runs normally while optstop reports what it would have stopped; and finally enable live stopping once the team is satisfied with the behavior.

For Inspect users, AISI says integration requires only a small amount of configuration, with randomized task ordering needed so early estimates are not biased by queue order. The open-source repository also exposes post-hoc and live workflows, multiple score types, convergence diagnostics and optional GPU acceleration.

Why it matters for frontier-model evaluation

The most important contribution is not simply lower benchmark cost. Adaptive allocation can let evaluation teams direct more of a fixed compute budget toward the parts of a test where uncertainty remains unresolved. That could matter for safety and capability evaluations where rare successes, long agent trajectories or repeated stochastic runs make fixed-budget testing expensive.

It also complements other work on evaluation integrity. Confidential or double-blind benchmark infrastructure helps protect tests from contamination, while tools such as optstop address how to spend evaluation compute once testing begins. Together, these approaches point toward evaluation systems that are both more trustworthy and more resource-aware.

The correct status is: optstop is available now as open-source software from AISI, with an accompanying paper and Inspect integration. Its reported savings are empirical results from AISI's test settings and should not be generalized as guaranteed reductions for every evaluation.