Senior Data Engineer

Senior Data Engineer - Responsibilities (7+ Years Exp) Build scalable data ingestion pipelines for relational systems and unstructured/semi-structured sources. Design and optimize lakehouse workloads using Apache Iceberg . Optimize table layouts, partitioning strategies, compaction, and schema evolution for Iceberg-based datasets. Implement best practices for versioning, time travel, incremental processing, and ACID guarantees .

Develop and tune Apache Spark (batch & streaming) for performant, scalable data processing. Work with AWS services (Glue, EMR, Lambda, Step Functions, S3) focusing on cost optimization and performance tuning . Implement and manage Kafka and Kafka streaming pipelines for large-scale real-time processing. Build and orchestrate data workflows using DBT and Airflow . Leverage AI/LLM automation to accelerate ETL/ELT development, validation, and debugging.

Implement observability—monitoring, logging, and alerting—to ensure platform stability and reliability. Participate in code reviews to ensure code quality and share best practices with the team. Required Skills 7 Years of relevant experience Strong SQL, Python, and Spark experience Deep experience ingesting and processing structured & unstructured data Strong knowledge of Apache Iceberg and lakehouse architectures Hands-on AWS experience with tuning and cost optimization Expertise with Kafka, Kafka Streaming, DBT, and Airflow Solid understanding of distributed systems and data engineering best practices