Overview
About the Role This is a senior, hands-on technical leadership role owning the strategy and systems that measure, improve, and scale training data for frontier AI agents. You will sit at the intersection of research and engineering, leading a team that defines what high-quality agent training data looks like and building the infrastructure to enforce that bar at scale. The work directly shapes the post-training data that aligns AI models to real-world tasks. What You'll Do Lead the data quality team in building evaluation systems across RL environments, synthetic data, benchmarks, and domain-specific workflows. Define data quality strategy by building QC systems, enforcing standards, and designing experiments to grade agent outputs. Develop methods for validating synthetic data at scale, including failure-mode analysis, task mutation checks, and trajectory auditing. Partner with research
Full job description
Full Job Description
About the Role
This is a senior, hands-on technical leadership role owning the strategy and systems that measure, improve, and scale training data for frontier AI agents. You will sit at the intersection of research and engineering, leading a team that defines what high-quality agent training data looks like and building the infrastructure to enforce that bar at scale. The work directly shapes the post-training data that aligns AI models to real-world tasks.
What You'll Do
-
Lead the data quality team in building evaluation systems across RL environments, synthetic data, benchmarks, and domain-specific workflows.
-
Define data quality strategy by building QC systems, enforcing standards, and designing experiments to grade agent outputs.
-
Develop methods for validating synthetic data at scale, including failure-mode analysis, task mutation checks, and trajectory auditing.
-
Partner with research engineers, domain experts, and data vendors to diagnose quality issues and improve data generation workflows.
-
Translate qualitative research insights into production systems: validation pipelines, dashboards, internal tools, and feedback loops.
-
Help build internal research taste around what makes agent training data realistic, learnable, diverse, reliable, and genuinely useful.
-
Mentor research engineers to maintain a high bar for technical rigor, clarity, and execution speed.
What We're Looking For
-
5+ years of experience in research or data quality engineering, specifically building systems for AI/ML data evaluation.
-
Demonstrated experience leading technical projects or teams in data quality or AI/ML evaluation, ideally on ambiguous, open-ended problems.
-
Advanced proficiency in Python, Docker, and Linux environments.
-
Deep, research-oriented understanding of AI evals and post-training, beyond surface-level agent frameworks.
-
Experience building QC systems, benchmarks, synthetic data pipelines, validation workflows, or model evaluation infrastructure.
-
Ability to reason carefully about what makes training data high-quality for AI agents, not just technically valid.
-
Experience translating research insights into production pipelines and internal tooling.
-
Ability to collaborate with domain experts and data vendors, capturing expert judgment and converting it into scalable review or generation systems.
-
Strong written communication skills, with the ability to explain methodology clearly to researchers, engineers, and external stakeholders.
-
Comfort designing metrics, experiments, and QA/QC processes independently.
-
Early-stage startup experience and the ability to move quickly in fast-paced, resource-constrained environments.
-
Detail-oriented mindset with a sharp eye for subtle inconsistencies and edge cases in data.
Compensation & Benefits
Salary range: $150,000 to $180,000 USD annually. Visa sponsorship is available.
Location
On-site in San Francisco, CA, United States. This role is not fully remote.
Tips for this job
Practical Job and Scholarship guidance. These tips do not replace official rules or create new eligibility requirements.
- Tailor the CV and application to the responsibilities and required skills stated on the official employer page.
- Use concrete evidence of relevant work, projects and measurable results rather than generic claims.
- Confirm location, work authorization, remote restrictions and sponsorship terms before applying.
- Apply through the original employer or official recruitment destination shown on this page.
Verification notes
Discovered directly from the employer’s public Ashby Job Postings API. The complete public role content and compensation metadata were normalized into safe candidate-facing sections.
Job and Scholarship is the discovery and verification layer. Confirm eligibility, dates, salary/funding and application instructions on the original source before submitting anything.
Clera (ashby) ↗Browse current Job and Scholarship listings from Clera (ashby) →
