Overview
Design and oversee evaluation systems that test LLM capabilities in the health domain, including internal benchmarking and regression testing that capture model accuracy, safety and utility - and make sure results are interpreted and clearly communicated to stakeholders. Guide the design of agentic, multi-step systems that combine prompt / context engineering, tool use and retrieval, and champion best practices for building and deploying them reliably at scale. Improve the internal tooling used to implement, run and analyse evaluations, and the data pipelines - dataset sourcing, curation and synthesis - that feed them. Required Bachelor's or higher degree in Computer Science or a related technical discipline, AND significant Python programming experience / machine learning research. Very strong proficiency with LLM evaluations - demonstrated experience designing, building and running eva
Full job description
Full Job Description
Design and oversee evaluation systems that test LLM capabilities in the health domain, including internal benchmarking and regression testing that capture model accuracy, safety and utility - and make sure results are interpreted and clearly communicated to stakeholders. Guide the design of agentic, multi-step systems that combine prompt / context engineering, tool use and retrieval, and champion best practices for building and deploying them reliably at scale. Improve the internal tooling used to implement, run and analyse evaluations, and the data pipelines - dataset sourcing, curation and synthesis - that feed them. Required Bachelor's or higher degree in Computer Science or a related technical discipline, AND significant Python programming experience / machine learning research. Very strong proficiency with LLM evaluations - demonstrated experience designing, building and running eval pipelines, curating and synthesising datasets, designing automated analyses, and explaining results to internal stakeholders. Very strong proficiency with LLM orchestration - deep, hands-on experience building with and around LLMs, including prompt / context engineering, tool use, harness engineering, retrieval and agentic, multi-step systems, and building tools to analyse and understand their performance. Proven engineering leadership - 8+ years of software engineering experience, including at least 3 years leading technical teams or projects as a tech lead and/or people manager - mentoring and developing engineers, and guiding a group to deliver high-quality results. Formal people-management experience is preferred; a strong technical lead ready to step into a TLM role will be considered. 0-to-1 experience with a bias towards shipping and learning while balancing a high-quality bar. Experience collaborating in cross-functional teams, working through ambiguity to deliver high-quality results, and a proven ability to contribute to a positive, inclusive work environment that fosters knowledge sharing and growth. Experience in healthcare technology, or experience in the health domain. Experience with data engineering - handling text dataset sourcing, curation and processing tasks at scale. Experience translating cutting-edge research into shipped products in a fast-paced, startup-like environment. Passionate about conversational AI and its deployment. Demonstrated written and verbal communication skills, with the ability to work closely with cross-functional teams including product managers, designers and other engineers. Passion for learning new technologies and staying up to date with industry trends, best practices and emerging patterns in AI.
Tips for this job
Practical JobOpportunity guidance. These tips do not replace official rules or create new eligibility requirements.
- Tailor the CV and application to the responsibilities and required skills stated on the official employer page.
- Use concrete evidence of relevant work, projects and measurable results rather than generic claims.
- Confirm location, work authorization, remote restrictions and sponsorship terms before applying.
- Apply through the original employer or official recruitment destination shown on this page.
Verification notes
Verified from public schema.org JobPosting structured data on the official source page. The complete published description, responsibilities, requirements and benefits were normalized when present; unstated facts were not inferred.
JobOpportunity is the discovery and verification layer. Confirm eligibility, dates, salary/funding and application instructions on the original source before submitting anything.
Apply through JobOpportunity →Browse current JobOpportunity listings from Microsoft Careers →
