MLPerf Adds End-to-End RAG Benchmark for Multi-Model AI Systems

MLPerf now benchmarks a full RAG system, not just one model

MLCommons announced on August 26, 2026 the first instance of a new MLPerf End-to-End Retrieval-Augmented Generation (RAG) Inference Benchmark. The important change is scope: instead of measuring one language model in isolation, the benchmark scores an entire RAG deployment pipeline that combines document processing, embedding, vector search, reranking and multiple language-model roles.

This is a released benchmark definition and reference implementation, not a new leaderboard result claiming one vendor or accelerator is fastest. The first instantiation focuses on MLPerf's Offline serving scenario. MLCommons says a Server scenario and further agentic extensions are future work.

Two workloads cover ingestion and question answering

The benchmark is split into two independent workloads. The ingestion pipeline, identified as e2e-rag-db, parses a frozen document corpus, chunks the text, generates semantic embeddings and builds a vector database. The QnA pipeline, e2e-rag-qna, then answers multi-hop questions by repeatedly retrieving evidence and reasoning over it.

For the document corpus, MLCommons uses the public FRAMES benchmark: 824 questions, 2,515 frozen Wikipedia HTML pages and roughly 107,000 passages after chunking. Each passage is 768 characters with a 32-character overlap. Embeddings are stored in a FAISS HNSW index.

The QnA stage is deliberately iterative. A query rewriter can produce up to three focused subqueries. Retrieved passages are reranked and graded for relevance, a sufficiency checker decides whether enough evidence has been collected, and the system can repeat this retrieve-and-reason loop for up to five hops before producing a grounded answer or returning Unknown.

The reference pipeline mixes several models and model sizes

MLCommons' published reference configuration uses several distinct components rather than one monolithic LLM. The embedder is intfloat/e5-base-v2, while ColBERTv2.0 handles reranking.

For language-model stages, the reference uses GPT-OSS-120B for query rewriting, sufficiency checking and final answer generation. GPT-OSS-20B is used as the document grader, and Llama 3.1 8B is used as the judge. MLCommons lists the two GPT-OSS models in MXFP4 reference precision, while the judge uses BF16.

That composition matters because real RAG systems often run different model sizes and numerical formats simultaneously. Optimizing such a pipeline can involve model placement, CPU-to-accelerator coordination, precision choices, memory placement and KV-cache-aware scheduling across several components. A single-model tokens-per-second benchmark cannot capture those system-level interactions.

Throughput is measured at the task level

The ingestion workload reports documents per second. The QnA workload reports tasks per second, where a task is one query completed end to end. MLCommons explains that tokens per second would be misleading because the pipeline invokes multiple models of different sizes and can take a variable number of reasoning hops.

The first benchmark version uses the Offline scenario, where all requests are available to the system together and can be scheduled for maximum throughput. MLCommons explicitly leaves the Server scenario for later rounds, so current discussion should not imply that this release already measures interactive online latency.

Accuracy gates keep performance submissions comparable

The reference implementation reports 35% final-answer accuracy over the full 824-query FRAMES set, with retrieval precision, recall and F1 of 75%, 70% and 69% respectively. A valid submission must reach at least 97% of the reference answer accuracy.

MLCommons also defines a compliance check, TEST09, that verifies mean answer-generator output length against the reference mean of 273.81 tokens within a plus-or-minus 10% range. The aim is to prevent a system from appearing faster simply by generating systematically shorter answers during a performance run.

These figures are reference-benchmark characteristics, not claims about the general quality of RAG systems or the underlying models outside this workload.

Why this matters for AI infrastructure

RAG is one of the most common ways enterprises connect foundation models to private or frequently changing information. Performance bottlenecks can appear in parsing, embedding, vector search, reranking, model serving, interconnects or the control loop that decides whether to retrieve again.

By measuring the whole pipeline, MLPerf gives hardware vendors, inference-engine developers and AI platform teams a common workload for optimizing those interactions while maintaining a defined accuracy floor. It also creates a clearer comparison target for systems that use heterogeneous accelerators or partition several models across one machine or cluster.

MLCommons also describes the benchmark as a stepping stone toward agentic AI benchmarking. The retrieve-reason-decide loop already resembles one of the core loops used by agents. Future versions could extend that structure with tool use, structured retrieval and greater autonomy.

For now, the correct status is: MLCommons has released the first End-to-End RAG benchmark definition and reference workload, with Offline throughput measurement available first; broader Server and agentic benchmark coverage is planned for later evolution.