Apple is where individual imaginations gather together, committing to the values that lead to great work. Every new product we build, service we create, or experience we deliver is the result of us making each other's ideas stronger.
Our team, part of Apple Services Engineering, is building the scientific foundation for how AI systems are evaluated across Apple. Our goal is to create the evidence that teams can act on: what to measure, whether a change helped, why a system failed, and whether it is ready to ship. We are seeking a Measurement Scientist to ensure that our evaluation methods are not just sophisticated, but scientifically valid and trustworthy.
In this role, you will apply apply state-of-the-art measurement methods, validity frameworks, experimental design, and statistical rigor to establish measurement standards for AI evaluation, ensuring that when we claim an evaluator measures "helpfulness" or "safety," it actually does. We are looking for individuals across a range of experience levels.
This role uniquely bridges measurement science and cutting-edge AI evaluation. This role uniquely bridges measurement science and cutting-edge AI evaluation. Our team is building AI tools and capabilities to help developers do evaluation better and faster. You will develop methods for validating these tools: making sure that they generate evaluation artifacts and workflows that give developers the right signals to make sound engineering decisions and build great products for Apple customers.
You will work on an interdisciplinary team with with other measurement scientists and ML researchers to solve new problems in AI evaluation. Your work will be both published at top measurement and ML venues and productionized into the evaluation SDK used across Apple. The successful candidate will have deep expertise in psychometrics or other measurement disciplines, or in experimental design and causal inference, with the ability to apply these principles to novel AI evaluation challenges.
Our team deliberately combines ML science and measurement science. We especially welcome people who bridge those areas, including computer scientists who specialize in measurement, and measurement scientists who have worked on ML problems. You will also work collaboratively with platform engineers and evaluation practitioners to translate measurement science into practical tools that scale across the organization.
Design validity frameworks for AI evaluation, ensuring that automated metrics, LLM-as-judge systems, and human evaluation protocols measure what they claim to measure across diverse contexts.
Develop and apply methods from measurement science to assessing the quality of benchmarks, for example drawing on frameworks like item response theory (IRT).
Create calibration and bias detection systems for automated evaluators, ensuring LLM-as-judge scores are interpretable, consistent, and free from systematic biases.
Build robust statistical tools for practitioners for sample-size planning, comparing system versions, quantifying uncertainty, controlling error rates, and visualizing data.
Connect offline evaluation to real-world outcomes: test whether offline score changes predict what users experience, design randomized audits, and develop methods for setting defensible ship thresholds that engineering teams can apply.
Establish measurement standards for evaluator transfer and generalization, including methods to quantify or predict when evaluators will maintain validity across domains, languages, or contexts.
Validate novel evaluation methods in collaboration with ML researchers, ensuring intelligent search algorithms discover statistically meaningful patterns, failure analyses estimate how often problems occur with uncertainty, and synthetic data generation produces representative samples.
Collaborate with platform engineers to productionize measurement methods into evaluation infrastructure, creating self-service tools for validity checking, reliability testing, and interpretable outputs (report cards, warnings, confidence metrics).
Publish research at top measurement venues and/or ML conferences (NeurIPS, ICML, ICLR), advancing both measurement science and AI evaluation.
Collaborate across disciplines with ML researchers developing novel methods, platform engineers building scalable infrastructure, and evaluation practitioners using these tools in production.
PhD in Computer Science or Machine Learning with a measurement or evaluation focus, Psychometrics, Educational Measurement, Quantitative Psychology, Statistics, Biostatistics, Econometrics, or equivalent research/work experience.
Deep expertise in modeling test or benchmark data (IRT, latent variable, or other response models) and construct validation, in benchmark design and validity for AI systems, or in experimental design and causal inference (surrogate outcomes, observational causal methods).
Strong statistical foundation including experimental design, power analysis, sampling theory, and uncertainty quantification.
Track record of designing and validating measurement instruments or experiments as demonstrated through publications or applied work.
Proficiency in Python (preferred) or R for statistical analysis, psychometric modeling, and method implementation.
Strong working knowledge of generative AI technology.
Excellent communication skills with the ability to explain complex measurement concepts to engineers, ML researchers, and non-technical stakeholders.
Research or experience applying measurement science to AI/ML evaluation, benchmark validity, automated scoring systems, or computational assessment.
Knowledge of modern ML evaluation challenges including LLM-as-judge, automated metrics, benchmark design, and agentic systems.
Publications at measurement venues or top ML conferences (NeurIPS, ICML, ICLR).
Expertise in computational social or behavioral science using generative AI.
Experience collaborating with engineers to turn research methods into production tools and scalable infrastructure.
Experience with online experimentation or with validating offline metrics against real-world outcomes (surrogate or proxy metrics).
Experience estimating rare events or making decisions when one kind of error costs far more than another.
Job description provided by Apple.
View the original posting