ML Platform Engineer, Distributed Training

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a ML Platform Engineer, Distributed Training based in the United States. Join a high-impact AI engineering team supporting healthcare and life sciences applications through large-scale foundation model training. As an ML Platform Engineer specializing in distributed training, you will work across GPU infrastructure and model training workloads on a Google Kubernetes Engine (GKE) platform powered by H200 and GB200 GPUs. You’ll collaborate with infrastructure engineers and research scientists to optimize training performance, diagnose complex failures, and improve multi-node execution at scale. Rather than focusing solely on cluster operations, you’ll take end-to-end ownership of the training stack, from workload scheduling and GPU networking to benchmarking and model evaluations. Your work will help scientists train models more efficiently while improving system reliability, scalability, and cost effectiveness. This role offers a hands-on engineering environment where technical depth, proactive problem-solving, and clear communication are essential.

Optimize the training platform: Partner with infrastructure teams to improve GKE-based training environments, including GPU node pools, job scheduling, queueing, resource quotas, and capacity management. Diagnose platform failures: Investigate underlying causes of infrastructure and training issues, resolve systemic problems, and implement durable fixes rather than relying on temporary workarounds. Tune GPU networking: Optimize multi-node GPU communication and training throughput using technologies and techniques such as RDMA, GPUDirect, NCCL, and GPU interconnect topology. Run distributed training workloads: Execute, scale, profile, and debug multi-node training jobs using PyTorch-based frameworks such as FSDP, DeepSpeed, Megatron, or NeMo, alongside Ray Train. Identify performance bottlenecks: Analyze compute utilization, memory consumption, input pipelines, and network communication to improve training efficiency and scalability. Design performance benchmarks: Develop and run benchmarks that measure throughput, scaling efficiency, and cost per training run, using the results to guide engineering decisions. Build model evaluation systems: Create and maintain evaluation harnesses that objectively measure the impact of changes to training recipes, models, and platform configurations. Improve observability and testing: Instrument training workloads and platform components with telemetry and monitoring, and maintain end-to-end pipeline tests that reflect real user workflows. Own technical outcomes: Proactively identify gaps, communicate project status and risks, propose solutions, and support the scientists who rely on the platform. Collaborate and communicate: Explain and defend architectural decisions to engineers and researchers, work effectively across teams, and adapt to evolving priorities and organizational structures. Requirements: Professional experience: At least 5 years of engineering experience, including recent hands-on work with GPU training infrastructure and large-scale model training. Python engineering: Strong production-level Python skills, including writing clean, maintainable code, automated tests, participating in code reviews, and following sound software development practices. Kubernetes and GKE: Practical experience with Kubernetes on Google Cloud, including GPU node pools, job scheduling and queueing systems such as Kueue, and resource quota and capacity management. Distributed training expertise: Direct experience running and debugging multi-node PyTorch training jobs and working with Ray Train. GPU networking and interconnects: Strong understanding of RDMA, GPUDirect, NCCL tuning, and multi-node GPU architectures involving H200 or GB200 hardware. Performance analysis: Demonstrated ability to design and execute training benchmarks, evaluate model performance, identify bottlenecks, and use empirical evidence to guide optimization. Technical depth: Ability to explain system architecture, implementation choices, and performance behavior in detail, demonstrating independent engineering judgment rather than relying on AI coding tools to supply the underlying reasoning. Ownership and communication: A track record of accountability, proactive problem-solving, transparent communication, and effective collaboration in demanding engineering environments. Preferred domain experience: Familiarity with healthcare or life sciences data, including genomics, clinical, or biomedical information, and experience working under HIPAA requirements. Additional preferred skills: Experience building observability and telemetry for training platforms, including metrics, tracing, alerting, and end-to-end pipeline testing. Benefits: Annual salary: $159,300–$219,245, with the final salary range determined by the assigned level and applicable factors such as experience, expertise, interview performance, and geographic location. Comprehensive health insurance: Healthcare coverage to support employee well-being. Paid time off: Vacation and general paid leave. Paid holidays: Paid time off for designated holidays. Sick leave: Leave to support health and recovery. Parental leave: Leave to support employees and their families. Bereavement leave: Support during times of loss. Retirement benefits: 401(k) plan with an employer match. Employee referral bonuses: Opportunities to receive bonuses for eligible referrals. Employment type: The position may be offered as full-time employment or contract work, as indicated in the original listing.