Anthropic collaboration introduces the Conceptual Reasoning Index for hard-to-verify AI reasoning

A benchmark for reasoning without easy feedback

Researchers from Redwood Research and Anthropic have introduced the Conceptual Reasoning Index (CRI), a benchmark suite aimed at a class of AI reasoning problems that are difficult to score through ordinary empirical or mathematical feedback. The motivation is especially relevant to AI safety and governance, where some decisions concern future systems, long time horizons or normative questions that may not have an immediately verifiable answer.

Three components feed the index

CRI currently aggregates three benchmarks. LMCA evaluates how well models judge conceptual arguments using a curated collection of position texts and expert-rated arguments. ACCoRD tests whether a model's stated probabilities and preferences remain logically consistent across related conceptual questions. DTBench evaluates decision-theoretic reasoning through handcrafted multiple-choice problems involving predictions of a model's own behavior or interactions with similar agents.

The researchers currently weight LMCA at 60% of the aggregate score, with ACCoRD and DTBench each contributing 20%. They say the composition may change as benchmarks saturate or new measurements are added.

Why it matters

The work highlights a gap between tasks where models can quickly test and refine answers and tasks where useful reasoning depends on argument quality, consistency and judgment. That distinction matters for AI risk analysis, governance design and long-range planning: strong coding or mathematically verifiable performance does not automatically show that a model can reason reliably when feedback is sparse.

As of the research snapshot dated August 10, 2026, the best measured CRI score remained materially below the authors' estimated ceiling, while scores across leading systems had continued rising since late 2024. The team plans to keep the public index updated as models and benchmarks evolve.

What to watch next

CRI is most useful as a complementary signal rather than a single ranking of overall intelligence. Future updates could show whether conceptual reasoning continues improving at the same pace, which components saturate first, and whether benchmark gains translate to real AI-safety and governance work. The primary research page and the maintained index should be treated as the authoritative sources for methodology and future score changes.