EPAM Systems Technology/Software

Lead Data Engineer (PySpark, statistics, MLflow)

Remote Full-time

We are looking for a Lead Data Engineer to join our team and take ownership of complex data engineering challenges, from migrating legacy codebases to building robust, production-grade pipelines. In this role, you will work independently on large-scale PySpark systems, apply statistical methods to real-world sensor data, and leverage MLflow to manage the full lifecycle of machine learning models. Responsibilities Migrate existing pandas, Koalas or pandas-on-Spark code to native PySpark implementations Decompose and refactor large legacy notebooks without relying on existing documentation Apply statistical techniques to detect outliers and events in sensor time series data Package and manage machine learning models using MLflow, including experiment tracking, Model Registry and versioning Implement model explainability techniques such as SHAP for existing models Develop unit and regression tests for PySpark pipelines using pytest and tools like assertDataFrameEqual or chispa Work independently on data engineering projects with minimal oversight Integrate AI coding assistants into daily workflows and critically review their output Utilize Window functions to optimize data transformation logic Requirements 5+ years of experience in data engineering with deep knowledge of PySpark and pandas Proven experience migrating pandas, Koalas or pandas-on-Spark code to native PySpark, including Window functions Capability to read and decompose large legacy notebooks without documentation, and to work independently on them Expertise in applied statistics, including outlier detection and event detection in sensor time series Proficiency in MLflow for experiment tracking, Model Registry and versioning, with skills in packaging existing models including explainability techniques like SHAP Skills in writing unit and regression tests for PySpark using pytest and assertDataFrameEqual or chispa Familiarity with using AI coding assistants in daily work and reviewing their output critically English proficiency at B2 level or higher Nice to have Knowledge of incremental processing on Delta, including MERGE and Change Data Feed, and Databricks job triggers Background in working with sensor or IoT data Databricks Machine Learning Professional certification German proficiency Romania We believe that the greatest strength of the company is its people.

EPAM is fully committed to help its employees to reach their full potential and achieve their professional goals through continues learning. With this in mind, we would like to introduce to you few of the many opportunities and services which we believe will help you expand your current knowledge: Full access to cutting-edge tools and technologies Competitive compensation depending on experience and skills All-around Social package: professional & soft skills training, medical & family care programs, sports Free English classes Unlimited access to LinkedIn learning solutions Continuous experience exchange with experts and professionals worldwide Friendly team and comfortable working environment Engineering, corporate, and social events within and outside the Company Flexible working schedule Opportunities for self-realization

Job description provided by EPAM Systems.
View the original posting