AWS and NVIDIA Show 75% GPU Reduction for Production ASR Inference

AWS and NVIDIA publish a production inference optimization pattern

AWS published a technical report on August 27, 2026, developed with NVIDIA and Heidi Health, showing how NVIDIA CUDA Multi-Process Service (MPS) and NVIDIA Triton Inference Server can increase GPU utilization for automatic speech recognition workloads on Amazon EC2.

This is not a new NVIDIA GPU or AWS managed AI service release. It is a production engineering pattern and benchmark for serving speech models more efficiently.

The utilization problem

AWS says a single ASR request in the tested NVIDIA Parakeet TDT 0.6B V2 workload used only about 15–20% of an NVIDIA L40S GPU's compute capacity.

With default CUDA time slicing, requests execute sequentially, leaving much of the GPU idle even when multiple processes are waiting.

Heidi Health's production system was using 16 GPU instances to maintain sub-second transcription latency during peak traffic.

What MPS changes

CUDA MPS allows multiple processes to share streaming multiprocessors concurrently instead of taking turns using the entire GPU.

In the published implementation, AWS combines MPS with Triton dynamic batching, a FastAPI gateway, CUDA graph warmup and health monitoring.

The system exposes an OpenAI Whisper-compatible transcription endpoint, making the serving layer easier to integrate with applications already using that API pattern.

Reported benchmark results

AWS reports that the selected production path, Triton plus MPS on g7e.4xlarge, sustained 92.1 requests per second per GPU at about 352 ms mean latency and 769 ms p99 latency at its chosen operating point.

The published SLA threshold was mean latency below 650 ms and p99 below 1,000 ms.

AWS says the resulting recommendation was four GPUs instead of 16, a 75% reduction in GPU infrastructure for the tested workload.

A more aggressive TensorRT + ONNX + MPS configuration reached 111.6 RPS per GPU and could reduce the recommended GPU count to two, an 88% reduction, but AWS notes the tradeoff of a longer deployment pipeline and recurring ONNX re-export.

These are workload-specific engineering results, not a promise that every speech or generative-AI system will see the same savings.

Reliability details matter

The post is valuable because it documents several issues that appear only under sustained production-style concurrency.

AWS describes an MPS-safe CUDA graph fallback to avoid recapture crashes when new tensor shapes appear, serialized model loading to avoid memory spikes, a sentinel-based health check for wedged CUDA processes, and dynamic batching tuned to latency limits.

The same broad pattern is described as model-agnostic and was also validated with NVIDIA Canary and OpenAI Whisper large-v3 checkpoints.

Why this matters

Inference cost is increasingly a systems problem rather than only a model-compression problem.

When individual requests use a small fraction of a GPU, concurrent execution can improve economics without changing model weights. MPS is one way to reclaim otherwise idle compute, while Triton provides scheduling and batching around the model.

For teams serving speech recognition, multimodal encoders or other under-utilized GPU workloads, the report provides a concrete blueprint and public implementation rather than only headline benchmark numbers.

The main caution is operational complexity: GPU sharing introduces new failure modes, and teams still need workload-specific load tests, p99 latency monitoring and isolation controls before adopting the same configuration in production.