Location: US, Indianapolis IN
At Lilly, the work is demanding because patients are waiting. We unite caring with discovery to help make life better for people around the world, knowing that every decision, every detail, and every day matters. Headquartered in Indianapolis, Indiana, our over 50,000 employees around the globe take on complex challenges to discover and deliver life-changing medicines, strengthen how health is understood and managed, and support the communities we serve. This is hard, urgent, selfless work—but it’s work worth doing. If you’re driven by purpose and ready to bring your best to work that truly matters for patients, we invite you to join us.
Purpose
The Associate Director, RWD Data Engineering is a hands-on technical role that builds and delivers the engineering vision for Lilly’s real-world data (RWD) infrastructure set by the Senior Director – RWD Engineering & Architecture. This individual designs, codes, tests, and deploys the scalable, cloud-native pipelines, data products, and analytic applications that allow HEOR, SDIA, Statisticians, Medical, and Clinical teams to generate evidence faster, more reproducibly, and at greater scientific depth than is possible through traditional vendor engagements.
This role requires deep, practical knowledge of secondary healthcare data, with a particular focus on administrative claims and electronic health records (EHR), and of the Databricks Lakehouse platform used to engineer it. The Associate Director turns raw RWD into validated, analysis-ready data products; automates data ingestion, quality checks, and product creation; and builds AI-enabled tools and analytic applications that put those data products directly in the hands of scientists.
This individual applies AI coding assistants and frontier AI models as an everyday part of software development, and is accountable for the correctness, performance, and reproducibility of the code they deliver. The ideal candidate is an expert builder who pairs strong engineering fundamentals with rigor in data quality and a passion for leveraging data to drive business outcomes.
This position reports to the Senior Director – Real World Data Engineering & Architecture within HEOR Central and works in close partnership with CIA, HEOR, SDIA, Statisticians, Medical, and Clinical teams.
Responsibilities
undefined
Design, develop, and deploy cloud-native data products and high-throughput data pipelines on Databricks that transform raw real-world data into scalable, reliable, analysis-ready assets supporting analytics, reporting, and evidence generation.
Leverage AI coding assistants and agentic development tools (e.g., Claude Code, GitHub Copilot, Cursor, Databricks Assistant) to write, refactor, test, and document Python, SQL, and R code, applying rigorous human review, testing, and validation to every AI-generated contribution.
Deliver assigned RWD data products through the full development lifecycle, from technical requirements and prototyping through production deployment and optimization, ensuring scalability, reliability, performance, and reproducibility within the established Databricks Lakehouse environment.
Build, optimize, and maintain ETL/ELT ingestion and transformation pipelines for large-scale secondary RWD, with a particular focus on administrative claims and complex EHR data, handling data volumes ranging from tens of millions to billions of records.
Build and maintain data assets within the established lakehouse architecture (e.g., medallion bronze/silver/gold patterns) using Databricks, Delta Lake, Unity Catalog, and Databricks Workflows to produce versioned, partitioned, and audit-ready data assets.
Write and maintain reusable, version-controlled transformation logic incorporating healthcare coding and terminology standards (e.g., ICD-10/ICD-9, NDC, RxNorm, SNOMED, CPT/HCPCS, LOINC) to produce domain-level datasets such as demographics, diagnoses, treatments, procedures, encounters, and labs.
Optimize SQL and distributed processing workloads (e.g., Databricks SQL and Spark jobs) for performance and compute cost across very large datasets, reading query plans and applying partitioning, liquid clustering/Z-ordering, predicate pushdown, caching, and other optimization strategies appropriate to analytical workloads.
Translate analytic and research requirements into reproducible data extraction and transformation logic, supporting cohort construction, temporal logic, and consistent reuse of RWD across teams.
Design, implement, and run data quality and validation frameworks for every data product, including automated checks for completeness, conformance, plausibility, and referential integrity; reconciliation of record counts and distributions against source; regression testing between data refreshes; and monitoring and alerting that surface issues before they reach analytical teams.
Apply the team’s version control and CI/CD standards in daily work, including Git branching, pull requests, peer code review, automated unit and data tests, and deployment via Databricks Asset Bundles, ensuring all code is version-controlled, tested, and deployable in a reproducible manner.
Build and deploy interactive analytic applications (e.g., Dash, R Shiny, Streamlit, Databricks Apps) that allow HEOR scientists, statisticians, and medical teams to explore data products, review cohort counts and data quality results, and self-serve common analyses.
Build AI-enabled solutions that operate directly on governed data products using frontier AI models (e.g., Anthropic Claude, OpenAI GPT, Google Gemini) and Databricks Model Serving, such as natural-language query of data products, AI agents that execute data workflows, and retrieval over data dictionaries, code lists, and documentation; evaluate these solutions for accuracy and ensure they meet Lilly’s privacy, security, and responsible-AI requirements.
Collaborate closely with data scientists, statisticians, and other technical partners to understand technical requirements, and contribute to documentation of RWD engineering standards, transformation templates, code list repositories, and pipeline performance guidelines.
Serve as the technical point of contact for assigned data sources and data products, advising collaborators on appropriate use, structural limitations, join strategies, and performance considerations; contribute to source-specific training materials for HEOR scientists, SDIA, and statisticians.
Contribute to an inclusive culture where producing and maintaining high-quality data is a core discipline.
Technical Skills
Core Engineering Skills
- Expert SQL optimization on Databricks SQL and Spark SQL (with Redshift, Snowflake, or BigQuery a plus): complex joins, window functions, query plan analysis, query tuning, and compute cost management on billion-row tables.
- Python fluency: pandas, PySpark, Polars for large-scale data manipulation; workflow orchestration with Apache Airflow, Prefect, or Dagster for production pipeline scheduling and monitoring — including Databricks Workflows for orchestrating multi-task jobs within the Lakehouse.
- Distributed computing: Apache Spark (PySpark), Dask, or Ray — ability to write, tune, and debug distributed jobs processing multi-terabyte datasets across partitioned cloud storage, including Databricks clusters with auto-scaling and spot instance optimization.
- Databricks expertise: hands-on, production development on Databricks (AWS, Azure, or GCP), including notebooks, Workflows, Delta Live Tables, Unity Catalog, SQL warehouses, and cluster configuration; experience with AWS (S3, Glue) or Azure (ADLS, ADF) object storage and services.
- Delta Lake / Apache Iceberg: time travel, schema evolution, upsert/merge operations, partition optimization — building versioned, ACID-compliant data assets at scale; Delta Lake experience ideally hands-on within the Databricks Lakehouse platform using Delta Live Tables (DLT) for declarative pipeline authoring.
- ETL/ELT tooling: AWS Glue, dbt, dbt-databricks, or equivalent for building tested, documented transformation pipelines with automated data quality checks; familiarity with Databricks Asset Bundles (DABs) for packaging and deploying notebooks, jobs, and DLT pipelines as code.
- Version control and DevOps: daily use of Git and GitHub/Azure DevOps (branching strategies, pull requests, code review, merge conflict resolution), CI/CD pipelines (GitHub Actions, Azure DevOps), and Docker, maintaining all code as version-controlled, testable, and deployable artifacts; experience deploying to Databricks via Asset Bundles, CLI, or REST API.
- Data modeling: building star/snowflake schemas, OMOP-compliant structures, and flat domain long files optimized for analytical workloads in a research context, materialized as managed Delta tables within Databricks Unity Catalog with appropriate access controls and lineage tracking.
Analytic Application Development
Building and deploying interactive analytic applications with Dash (Plotly), R Shiny, Streamlit, or Databricks Apps, connected to Databricks SQL warehouses and Unity Catalog with appropriate authentication and access controls.
Front-end fundamentals sufficient to deliver usable, performant applications for scientific users, including caching, pagination, and efficient query patterns against large tables.
R programming: advanced R development for data engineering and analytics, including building and deploying production R Shiny dashboards; writing and maintaining internal R packages that standardize how teams connect to, query, and analyze data products (e.g., shared connection functions to Databricks, cohort-building utilities, and reusable analytic functions); and applying good package development practices such as roxygen2 documentation, testthat unit tests, version control, and renv for reproducible environments.
Data Quality & Validation
Designing automated data quality frameworks using tools such as Delta Live Tables expectations, Great Expectations, dbt tests, or OHDSI Data Quality Dashboard (DQD), with results tracked across data refreshes.
Validating claims and EHR data products against source, vendor documentation, and published benchmarks, including record reconciliation, distribution and trend checks, and investigation and resolution of data anomalies.
Healthcare RWD — Claims and EHR
Administrative claims: deep, hands-on working knowledge of medical and pharmacy claims from sources such as Optum Clinformatics, IQVIA PharMetrics, and Komodo, including enrollment/eligibility tables, revenue codes, place-of-service codes, inpatient vs. outpatient claim splitting, drug identification at NDC and GPI level, and known structural quirks of each source.
Complex EHR data: HL7 FHIR resources, Epic/Cerner/Truveta data models, clinical note schemas, problem list hierarchies, medication order and administration tables, lab result normalization, and vital sign time series — including semi-structured and nested JSON/XML from EHR exports and FHIR APIs.
Healthcare terminology and ontology mapping: ICD-10-CM/PCS, ICD-9-CM, NDC, RxNorm, SNOMED CT, CPT-4, HCPCS, LOINC, ATC — building, versioning, and governing code list repositories joined to raw tables to produce concept-labeled analysis-ready datasets.
OMOP CDM: transforming source claims and EHR data to OMOP v5.x including vocabulary loading (Athena), ETL specification documentation, and Achilles/DQD data quality checks.
Phenotyping and cohort construction: translating clinical study protocols into reproducible extraction logic — index date derivation, washout periods, time-varying covariates, censoring — suitable for pharmacoepidemiology and HEOR studies.
Regulatory and privacy frameworks: HIPAA-compliant data handling, de-identification standards (Safe Harbor, Expert Determination), DUA compliance, and audit trail requirements for FDA-grade RWE submissions.
AI Fluency
AI-assisted software development: expert, daily use of AI coding assistants and agentic coding tools (e.g., Claude Code, GitHub Copilot, Cursor, Databricks Assistant) to write, refactor, debug, test, and document production code, with strong judgment on reviewing and validating AI-generated code.
Building with frontier models: hands-on experience integrating frontier AI model APIs (e.g., Anthropic, OpenAI, Google, Databricks Foundation Model APIs) into solutions that operate on data products, including prompt engineering, tool use/function calling, structured outputs, retrieval-augmented generation (RAG), and agentic workflows.
Evaluating AI solutions: designing test sets and evaluation methods to measure the accuracy and reliability of AI outputs (e.g., natural-language-to-SQL correctness, extraction accuracy), and implementing guardrails appropriate for patient-level data.
Implementing AI-powered data profiling and anomaly detection to automate quality checks across large ingestion runs and surface issues before they reach analytical teams.
Familiarity with MLOps tooling (MLflow, Databricks Model Serving, Vector Search) for versioning, deploying, and monitoring AI models integrated with data pipelines; experience applying NLP or LLMs to extract structured data from unstructured clinical notes a plus.
Minimum Qualification Requirements
- Bachelor’s in Computer Science, Applied Mathematics, Engineering, Statistics, Information Technology, Bioinformatics.
- Minimum 5 years' experience of hands-on data engineering experience with a demonstrated focus on healthcare or life sciences RWD, including direct work with large administrative claims and/or EHR datasets.
- Minimum of 1 year experience of hands-on, production development on the Databricks platform using Python (PySpark), SQL, and Delta Lake, applied directly to real-world healthcare data at scale.
- Minimum of 1 year experience designing and implementing data quality and validation checks for large clinical datasets.
- Minimum of 1 year of experience using AI coding assistants to develop production software, and experience building at least one solution using frontier AI model APIs.
Other Information/Additional Preferences
Experience with OMOP CDM transformation, vocabulary management (Athena), and federated analytics networks (OHDSI, PCORnet, Sentinel).
Master’s degree in Computer Science, Applied Mathematics, Engineering, Statistics, Information Technology, Bioinformatics.
Strong problem-solving skills, attention to detail, and ability to work effectively in matrixed, cross-functional teams with both technical and scientific partners.
Experience with distributed computing frameworks (Spark, Dask) for large-scale RWD processing.
Experience building and deploying analytic applications using Dash, R Shiny, Streamlit, or similar frameworks.
Strong understanding of healthcare coding standards (ICD-10, NDC, RxNorm, SNOMED CT, CPT, LOINC), claims and EHR data structures, and major RWD vendors and platforms (e.g., Merative MarketScan, Optum, IQVIA, Komodo, Truveta, HealthVerity).
Daily proficiency with Git-based version control, including branching, pull requests, and code review.
Advanced SQL optimization skills on Databricks SQL / Spark SQL, including query plan analysis, query tuning, and compute cost management.
Demonstrated ability to build production-grade ingestion pipelines for multi-terabyte secondary healthcare datasets, including administrative claims and complex EHR data.
Experience with data pipeline observability and quality tooling (e.g., Great Expectations, Monte Carlo, Lakehouse Monitoring).
Familiarity with FDA RWE Framework guidance, EMA RWD guidance, and audit-readiness requirements for observational studies.
Knowledge of Agile/Scrum methodologies and project management tools such as JIRA
Databricks certification (e.g., Data Engineer Associate or Professional, Generative AI Engineer Associate).
Experience with Model Context Protocol (MCP) or similar frameworks for connecting AI models to data and tools.
A portfolio of code (e.g., GitHub repositories) or deployed analytic applications demonstrating hands-on engineering work.
Recruitment open across sectors: pharma/biotech data teams, RWD vendors (Truveta, Komodo, Optum, IQVIA, HealthVerity), cloud providers (AWS, Google, Microsoft), genomics platforms (Illumina, DNAnexus, Broad Institute), or academic health systems with large-scale RWD programs.
NOTE: This job description is intended to provide a general overview of the job requirements at the time it was prepared. The job requirements of any position may change over time and may include additional responsibilities not specifically described in the job description. For GMP purposes, the job description should be updated for significant changes. As always, you should consult with your supervisor regarding your actual job responsibilities and any related duties that may be required for the position.
Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form (https://careers.lilly.com/us/en/workplace-accommodation) for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.
Lilly is proud to be an EEO Employer and does not discriminate on the basis of age, race, color, religion, gender identity, sex, gender expression, sexual orientation, genetic information, ancestry, national origin, protected veteran status, disability, or any other legally protected status.
Our employee resource groups (ERGs) offer strong support networks for their members and are open to all employees. Our current groups include: Africa, Middle East, Central Asia (AMECA), Black Employees at Lilly (BE@Lilly), Chinese Culture Network (CCN), EnAble, Evolve, Lilly Indian Network (LIN), Organization of Latinx at Lilly (OLA), Pride (LGBTQ+ Allies), Veterans Leadership Network (VLN) and Women’s Initiative for Leading at Lilly (WILL).
Actual compensation will depend on a candidate’s education, experience, skills, and geographic location. The anticipated wage for this position is
$136,500 - $200,200
Full-time equivalent employees also will be eligible for a company bonus (depending, in part, on company and individual performance). In addition, Lilly offers a comprehensive benefit program to eligible employees, including eligibility to participate in a company-sponsored 401(k); pension; vacation benefits; eligibility for medical, dental, vision and prescription drug benefits; flexible benefits (e.g., healthcare and/or dependent day care flexible spending accounts); life insurance and death benefits; certain time off and leave of absence benefits; and well-being benefits (e.g., employee assistance program, fitness benefits, and employee clubs and activities).Lilly reserves the right to amend, modify, or terminate its compensation and benefit programs in its sole discretion and Lilly’s compensation practices and guidelines will apply regarding the details of any promotion or transfer of Lilly employees.
#WeAreLilly
Additional Locations: US, Remote
Free tool / Resume tailor
Tailor your resume for this job
Paste your resume and get 3 to 8 of your own lines rewritten in the words this posting uses, with the reason for each change. The job is already filled in, and you don't need an account.
Job description provided by Eli Lilly.
View the original posting