Job
Source description available
Source: Clera (ashby) · A retrieval date records when our system last obtained the source record. It does not guarantee the vacancy is still open or that every detail has been independently checked.
Description from the source
The source description is formatted below for discovery. The provider owns the original wording and may change its requirements or close applications.
EmploymentFull-time
Overview
ABOUT THE ROLE
Full job description
ABOUT THE ROLE Join an AI team building systems that help teams identify and address failures in production AI agents. You will own the intelligence behind failure detection, evaluation, and proposed fixes, helping make agents more reliable and trustworthy. WHAT YOU'LL DO
- Detect subtle agent failures, including omissions, compliant but incorrect answers, and patterns across large volumes of traces.
- Use user behavior signals such as rephrasing, abandonment, and retries to identify issues that users do not explicitly report.
- Develop evaluation methods to measure detection quality when labeled ground truth is unavailable.
- Build trustworthy patch generation workflows that reproduce failures, verify fixes, and avoid proposing unreliable changes.
- Improve the cost-effectiveness of detection by choosing appropriate models and methods for each task.
- Review production customer traces regularly to find opportunities for improvement. WHAT WE'RE LOOKING FOR
- Applicants with 0 to 10 years of experience are welcome, including new graduates and candidates without a degree.
- Experience shipping products or systems that people use.
- Hands-on experience with production LLMs, evaluations, LLM-as-judge approaches, and embeddings.
- Good judgment in assessing research results and distinguishing meaningful improvements from noise, including when ground truth is unavailable.
- Experience operating AI agents in production is a plus.
- Must be authorized to work in the United States without visa sponsorship. COMPENSATION & BENEFITS Salary range: $120,000 to $200,000 USD annually. Visa sponsorship is not available. LOCATION On-site in San Francisco, United States.
Tips for this job
Practical JobOpportunity guidance. These tips do not replace official rules or create new eligibility requirements.
- Tailor the CV and application to the responsibilities and required skills stated on the official employer page.
- Use concrete evidence of relevant work, projects and measurable results rather than generic claims.
- Confirm location, work authorization, remote restrictions and sponsorship terms before applying.
- Apply through the original employer or official recruitment destination shown on this page.
Original authoritative source
JobOpportunity.info helps you discover and organize source listings. Confirm eligibility, dates, salary/funding and application instructions on the original source before submitting anything.
Apply through JobOpportunity →