AWS AgentCore Evaluations Now Scores Agents Across Major Frameworks

Agent evaluation is becoming framework-independent

AWS announced on August 26, 2026 that Amazon Bedrock AgentCore Evaluations can evaluate agents built with multiple frameworks through a common telemetry contract rather than requiring a framework-specific evaluation stack.

AWS lists direct support for Strands Agents, LangGraph, the OpenAI Agents SDK, LlamaIndex, Google ADK and the Claude Agent SDK. The service can also work with additional frameworks when their instrumentation follows recognized OpenTelemetry GenAI or OpenInference conventions.

OpenTelemetry is the compatibility layer

AgentCore Evaluations reconstructs an agent session from OpenTelemetry traces and CloudWatch event records. It primarily needs three span roles: the top-level agent invocation, individual model-inference calls and tool-execution spans. Retrieval, reranking, memory and guardrail spans can remain in the trace as additional context without requiring special evaluator configuration.

This matters because agent teams often combine multiple orchestration frameworks and model providers. A shared telemetry layer makes it possible to compare agent quality without first normalizing every SDK into a proprietary evaluation format.

Built-in and custom evaluators can use the same traces

AWS says the same reconstructed session can be scored with built-in evaluators such as GoalSuccessRate, Correctness and Helpfulness, as well as custom LLM-as-a-judge evaluators. The service supports both on-demand and online evaluation modes.

On-demand evaluation is designed for regression tests and CI/CD, where teams can provide expected responses, tool trajectories or assertions. Online evaluation samples production traffic continuously and writes results to CloudWatch, although evaluators requiring ground-truth reference data are not suitable for that live mode.

Instrumentation details still matter

Framework-agnostic does not mean zero integration requirements. AWS documents that spans need recognizable instrumentation scope names, a consistent session identifier and access to message content. Missing buffered telemetry can also cause evaluation failures, which is why AWS recommends flushing both tracing and logging providers before an AgentCore runtime invocation finishes.

Custom instrumentation must intentionally follow the supported conventions. A private tracing scope with arbitrary naming will not automatically be interpreted just because its payload looks similar.

Why this release is useful

Production agent systems increasingly mix orchestration libraries, model providers and tools. A common evaluation layer lets platform teams run the same quality gates across those stacks, compare regression scores across implementations and monitor live behavior without rewriting the evaluator for every agent framework.

The practical implication is that observability standards are becoming part of the agent platform contract: if an agent emits interoperable telemetry, evaluation can become portable too.