Overview
About the Role Join an AI/ML team building tools to identify and address failures in production AI agents. This role owns the intelligence behind failure detection, evaluation, and proposed fixes, helping make agents more reliable for real-world use. What You'll Do Detect subtle agent failures, including omissions, incorrect responses, and patterns across large volumes of traces. Turn user behavior such as rephrasing, abandonment, and retries into evidence of potential failures. Build evaluation methods that measure detection quality, including in situations without ground-truth labels. Verify that proposed fixes reproduce and resolve failures, and avoid recommending changes when evidence is weak. Improve the cost efficiency of detection by choosing appropriate models and methods for each task. Review production traces regularly to identify problems and guide product improvements. What W
Full job description
Full Job Description
About the Role
Join an AI/ML team building tools to identify and address failures in production AI agents. This role owns the intelligence behind failure detection, evaluation, and proposed fixes, helping make agents more reliable for real-world use.
What You'll Do
-
Detect subtle agent failures, including omissions, incorrect responses, and patterns across large volumes of traces.
-
Turn user behavior such as rephrasing, abandonment, and retries into evidence of potential failures.
-
Build evaluation methods that measure detection quality, including in situations without ground-truth labels.
-
Verify that proposed fixes reproduce and resolve failures, and avoid recommending changes when evidence is weak.
-
Improve the cost efficiency of detection by choosing appropriate models and methods for each task.
-
Review production traces regularly to identify problems and guide product improvements.
What We're Looking For
-
Demonstrated AI/ML engineering experience, with 0 to 10 years of experience considered.
-
Experience shipping products that people use and working as an individual-contributor engineer.
-
Hands-on experience with LLMs in production, including evaluations, LLM-as-judge, embeddings, and deciding when to use smaller models or no model.
-
Experience running AI agents in production and handling their failures.
-
Strong research judgment and the ability to distinguish meaningful improvements from noise, including when direct ground truth is unavailable.
Compensation & Benefits
Base salary range: $120,000 to $200,000 annually. Visa sponsorship is not available. The role requires authorization to work in the United States without visa sponsorship.
Location
On-site in San Francisco, United States.
Tips for this job
Practical JobOpportunity guidance. These tips do not replace official rules or create new eligibility requirements.
- Tailor the CV and application to the responsibilities and required skills stated on the official employer page.
- Use concrete evidence of relevant work, projects and measurable results rather than generic claims.
- Confirm location, work authorization, remote restrictions and sponsorship terms before applying.
- Apply through the original employer or official recruitment destination shown on this page.
JobOpportunity.info helps you discover and organize source listings. Confirm eligibility, dates, salary/funding and application instructions on the original source before submitting anything.
Apply through JobOpportunity →