AWS Adds Managed Ray Workflows to SageMaker HyperPod for Training and Inference
AWS has added a managed Ray experience to SageMaker HyperPod on EKS, including Studio-based cluster management, Ray observability, resilient training, tiered checkpointing and Ray Serve inference.
SageMaker HyperPod gets a managed Ray experience
AWS has announced new Ray capabilities for Amazon SageMaker HyperPod, bringing managed cluster creation, interactive development, observability, resilient training, and accelerated inference into the HyperPod experience on Amazon EKS.
The launch was announced on August 24, 2026 and is available in AWS Regions where SageMaker HyperPod EKS is supported.
Ray remains open source, and AWS says the integration works with KubeRay and standard Ray APIs so existing scripts can continue to run without being rewritten around a proprietary execution API.
What changes for Ray users
Running Ray on Kubernetes normally requires teams to manage cluster manifests, container dependencies, port forwarding, observability plumbing, and recovery behavior.
With the new HyperPod integration, SageMaker Studio can create and manage Ray clusters directly. Users can open Ray Dashboard and Amazon Managed Grafana views, attach JupyterLab or Code Editor workspaces, submit distributed jobs, and configure hung-job detection from the same environment.
Advanced users can still edit the underlying Kubernetes manifest, and the KubeRay operator remains responsible for RayCluster, RayJob, and RayService resources.
AWS also provides a toolkit-for-ray-on-sagemaker-ai package for remote job submission from Studio, local developer machines, or CI/CD systems using standard Ray job commands with an IAM-authenticated SageMaker-aware address.
Interactive development without rebuilding the cluster
HyperPod JupyterLab and Code Editor spaces can attach to an active Ray cluster as zero-compute worker nodes.
A notebook can connect using normal Ray initialization, and developers can use Ray runtime environments to inject Python dependencies without rebuilding the full cluster image for each change.
That creates a shorter path from interactive prototyping to distributed GPU execution: the same notebook can begin with a small worker count and then scale Ray Train workloads across additional GPU workers.
Observability is included
The HyperPod Observability EKS add-on now discovers Ray head and worker pods, scrapes metrics, and provisions prebuilt Grafana dashboards for Ray Core, Ray Data, Ray Train, and Ray Serve.
AWS says this removes the need to manually create Prometheus monitoring resources, configure scrape targets, wire IAM signing, and import Ray dashboard JSON.
The Ray metrics are displayed alongside HyperPod infrastructure views such as GPU, EFA, and task-governance dashboards.
Resilient distributed training
The new Ray workflow integrates with HyperPod’s infrastructure recovery features.
AWS describes three layers:
- Automatic node recovery — unhealthy nodes can be replaced while Ray reschedules workers.
- Hung-job detection — a HyperPod agent monitors training signals and can flag or cancel stalled jobs.
- Tiered checkpointing — checkpoints are written locally and asynchronously uploaded to Amazon S3, with HyperPod tiered storage used for faster recovery when available.
Ray jobs still need appropriate retry and checkpoint-resume logic. HyperPod handles infrastructure health and monitoring, but it cannot reconstruct application state that the training code never saved.
For large GPU clusters, automatic hang detection can be important because a distributed collective can stall while GPUs remain allocated and continue generating cost.
Ray Serve and long-context inference
The launch also supports Ray Serve workloads on HyperPod EKS.
A new JumpStart model loader in AWS’s Ray toolkit can pull model weights from the SageMaker JumpStart catalog and deploy them to Ray Serve without a separate manual model-download workflow.
For long-context serving, AWS integrates Managed Tiered KV Cache. Attention key-value vectors can use CPU memory as one tier and HyperPod Tiered Storage as another, allowing cache data to be shared beyond a single instance.
AWS says this can reduce time to first token for multi-turn and long-document workloads with minimal changes to Ray Serve deployments.
Why this matters for AI infrastructure teams
Ray is widely used to scale Python workloads for data processing, distributed training, reinforcement learning, evaluation, and model serving. Kubernetes provides a flexible substrate, but production operations can become complex when teams must assemble scheduling, observability, recovery, notebooks, and model-serving components themselves.
HyperPod’s new integration is AWS’s attempt to make that operational layer more managed while retaining standard Ray and KubeRay interfaces.
The most important trade-off is architectural: teams gain integrated AWS operations and recovery while still running on EKS and using open Ray APIs. That can reduce migration friction for existing Ray code, but users remain responsible for AWS infrastructure cost, cluster sizing, security configuration, and application-level checkpoint correctness.
Availability
The new Ray capabilities are released and available today, according to AWS, in all Regions where SageMaker HyperPod with EKS is supported. This is a production platform capability, not a future preview announcement.
This article is built from the source material below. Open the originals for full context and the latest updates.
