Data Engineer

Data Engineer – Responsibilities (4–7 Years Exp) Develop and maintain scalable data ingestion and processing pipelines using SQL, Spark, and Python/Scala. Develop and optimize Apache Spark batch and streaming jobs for large-scale data processing. Design and implement data models including Dimension and Fact tables , with clearly defined grain, relationships, and measures. Implement Slowly Changing Dimensions (SCD) and appropriate incremental processing strategies.

Work with AWS data engineering and serverless services such as Glue, S3, Lambda, EMR, and Step Functions. Develop and maintain reliable, scalable data pipelines with appropriate error handling, logging, monitoring, and data quality checks . Apply data engineering fundamentals including partitioning, incremental processing, schema evolution, data quality, and pipeline reliability . Work with Apache Iceberg , including table design, partitioning, schema evolution, and incremental processing.

Develop data transformation workflows using DBT and orchestrate pipelines using Airflow . Troubleshoot and optimize data pipelines for performance, scalability, and cost efficiency . Participate in code reviews, testing, debugging, and continuous improvement of data engineering solutions. Required Skills 4–7 years of relevant Data Engineering experience. Strong SQL skills — SQL is a core requirement. Strong hands-on experience with Apache Spark and Python or Scala .

Hands-on experience with AWS data engineering and serverless services , particularly AWS Glue and S3. Good understanding of Data Modelling , including: Dimension and Fact table design Defining appropriate Fact grain Dimension-to-Fact relationships Implementing SCD Type 1 / Type 2 and other appropriate SCD patterns Strong understanding of data engineering fundamentals including partitioning, incremental processing, schema evolution, data quality, and pipeline reliability .

Good understanding of distributed data processing and Spark performance optimization. Additional Responsibilities / Good to Have Experience with Apache Iceberg and lakehouse architectures. Experience with DBT for data transformation and modelling. Experience with Airflow for workflow orchestration. Kafka / Kafka Streaming experience is a good to have. Exposure to AI/LLM tools for development, debugging, testing, and documentation.