Overview
About the Role This role sits at the intersection of technical program management, data operations, and vendor coordination for an early-stage AI/ML infrastructure company. You will own complex data and evaluation programs for frontier AI labs and internal teams, turning ambiguous technical asks into well-structured, executable programs. Your work directly improves the quality of LLM and agent evaluation at scale. What You'll Do Own data and evaluation programs end-to-end, from initial scoping and requirements gathering through production, quality assurance, delivery, and retrospective. Translate ambiguous requests from AI labs and internal teams into clear specifications with defined milestones, owners, dependencies, and acceptance criteria. Maintain and improve data quality procedures using quantitative signals and qualitative inspection, including analysis of task coverage, difficulty
Full job description
Full Job Description
About the Role
This role sits at the intersection of technical program management, data operations, and vendor coordination for an early-stage AI/ML infrastructure company. You will own complex data and evaluation programs for frontier AI labs and internal teams, turning ambiguous technical asks into well-structured, executable programs. Your work directly improves the quality of LLM and agent evaluation at scale.
What You'll Do
-
Own data and evaluation programs end-to-end, from initial scoping and requirements gathering through production, quality assurance, delivery, and retrospective.
-
Translate ambiguous requests from AI labs and internal teams into clear specifications with defined milestones, owners, dependencies, and acceptance criteria.
-
Maintain and improve data quality procedures using quantitative signals and qualitative inspection, including analysis of task coverage, difficulty, and reward distributions.
-
Manage external vendors throughout the delivery lifecycle and improve delivery pipelines at scale.
-
Streamline data production logistics by improving workflows, documentation, automation, and cross-functional handoffs.
-
Partner with research, platform, and marketplace teams to translate learnings and feedback into product improvements.
What We're Looking For
-
3+ years owning complex technical programs in AI/ML, data operations, support engineering, applied research, or similarly cross-functional environments.
-
Demonstrated ability to translate ambiguous technical requirements into executable specifications with clear milestones, owners, dependencies, and acceptance criteria.
-
Strong quantitative and qualitative judgment about data, with the ability to move between aggregate metrics and individual examples to assess dataset quality.
-
Experience maintaining and improving complex data quality procedures using both quantitative metrics and qualitative inspection methods.
-
Experience managing external vendors and scaling delivery pipelines.
-
Background supporting frontier AI labs, technical enterprise customers, or research teams with urgent and evolving requirements.
-
Experience with RLHF, reinforcement-learning environments, agent evaluations, human-in-the-loop data pipelines, annotation workflows, or expert data collection.
-
Strong written and verbal communication skills, with the ability to turn complex technical information into clear decisions and next steps.
-
Early-stage startup experience and comfort working independently in fast-paced environments.
Compensation & Benefits
Competitive compensation package including equity. Full benefits include 100% employer-covered medical, dental, and vision insurance, a 401k, commuter benefits, and unlimited access to leading AI productivity tools. Visa sponsorship and relocation support are available for strong full-time candidates.
Location
On-site in San Francisco, CA or Singapore. Remote candidates who can maintain 70 to 80% time zone overlap with either office location are also welcome to apply.
Tips for this job
Practical Job and Scholarship guidance. These tips do not replace official rules or create new eligibility requirements.
- Tailor the CV and application to the responsibilities and required skills stated on the official employer page.
- Use concrete evidence of relevant work, projects and measurable results rather than generic claims.
- Confirm location, work authorization, remote restrictions and sponsorship terms before applying.
- Apply through the original employer or official recruitment destination shown on this page.
Verification notes
Discovered directly from the employer’s public Ashby Job Postings API. The complete public role content and compensation metadata were normalized into safe candidate-facing sections.
Job and Scholarship is the discovery and verification layer. Confirm eligibility, dates, salary/funding and application instructions on the original source before submitting anything.
Clera (ashby) ↗Browse current Job and Scholarship listings from Clera (ashby) →
