Lead AI/ML Engineer (LLM Quality)
About the role
We are hiring Lead AI/ML Engineer (LLM Quality) to design and develop robust evaluation systems for LLM inference quality. You will be leading efforts in designing autoraters, benchmarking pipelines, and analysis tools to quantify quality, bias, latency, and cost across multiple model vendors.
You'll work directly with the founding team. As an early hire, you'll have high autonomy and visibility, and play a central role in product, research, and engineering strategy.
Key Responsibilities
Design and implement scalable evaluation frameworks for LLMs.
Lead data collection, prompt tuning, and benchmarking efforts across model APIs.
Build and maintain internal autorating pipelines that mimic human evaluation quality.
Collaborate on development of a unified dashboard for inference quality metrics (latency, cost, token-level quality, robustness).
Contribute to strategic product decisions and roadmap prioritization.
Collaborate with external stakeholders and partners for pilot deployments.
Minimum Qualifications
Master's or PhD in Computer Science, Machine Learning, Applied Mathematics, or a related field.
5+ years of professional experience in AI/ML engineering, including hands-on work with LLMs or NLP systems.
Strong programming skills in Python and ML frameworks.
Solid understanding of LLM inference principles, prompt engineering, and transformer internals.
Experience with cloud-based infrastructure.
Preferred Qualifications
Previous experience at an AI startup or a major technology company involved in LLM frontier development.
Publications or open-source contributions in LLMs.
Demonstrated experience in model evaluation, benchmarking, or system diagnostics.
Experience building internal developer tools or observability systems.
What We Offer
Highly competitive compensation.
Attractive stock option plan.
Full Swiss employee benefits (AHV/IV, pension plan, accident insurance, etc.).
Access to compute and tooling resources from top-tier partners.
Opportunity to shape core product strategy.
A collaborative, impact-driven team culture in the heart of Zurich.