Lead Big Data Engineer

Data Engineer

The Opportunity

As a Data Engineer, you will be responsible for designing, building, and maintaining scalable data solutions that support enterprise-wide analytics and business intelligence initiatives. The role requires deep expertise in Big Data technologies, cloud platforms, and modern data engineering practices. In addition to technical leadership, you will play a key role in mentoring and developing Data & Analytics talent while driving delivery excellence across complex programs.

Key Responsibilities

• Design, develop, and maintain robust ETL/ELT pipelines on Azure cloud and on-premises platforms for batch and real-time data processing.
• Build scalable data ingestion and processing frameworks to handle structured, semi-structured, and unstructured datasets.
• Solve complex business problems by leveraging data engineering best practices and advanced technologies.
• Monitor, optimize, and troubleshoot data pipelines to ensure high availability, reliability, performance, and scalability.
• Ensure adherence to data quality, governance, security, privacy, and compliance standards.
• Collaborate with business stakeholders, architects, data scientists, governance teams, DevOps teams, and analytics partners to deliver business value.
• Lead solution design discussions, technical estimations, architecture reviews, and implementation strategies.
• Provide technical guidance and mentorship to engineering teams while fostering continuous learning and capability development.
• Drive adoption of engineering standards, reusable frameworks, and best practices across programs.

Primary Requirements

A. Technical Skills

• Strong experience with SQL, Python, and/or Scala.
• Hands-on expertise in Big Data technologies including Apache Spark, Hadoop, and Hive.
• Experience designing and developing enterprise-scale ETL/ELT pipelines.

Expertise in Azure Data Engineering ecosystem

Azure Data Factory (ADF)

Azure Synapse Analytics

Azure Databricks

Azure Event Hub

Azure Functions

• Experience with NoSQL and Distributed Databases such as Cosmos DB and HBase.
• Strong understanding of Data Warehouses, Data Lakes, and Data Marts.
• Experience implementing Medallion Architecture and modern data design patterns.
• Expertise in data performance tuning, optimization, and quality validation.
• Experience building both real-time and batch processing pipelines, including Spark Streaming.

B. Additional Requirements

• Excellent communication and stakeholder management skills.
• Strong analytical and structured problem-solving capabilities.
• Proven experience in solution architecture, estimation, and technical leadership.
• Ability to effectively communicate technical concepts through storytelling and presentations.
• Experience managing relationships with business partners and senior stakeholders.
• Demonstrated people leadership experience, including coaching, mentoring, talent development, and engagement.
• Good to Have
• Experience with DevOps and CI/CD practices using Git and Azure DevOps.
• Unix/Linux shell scripting expertise.
• Experience with MongoDB and Apache NiFi.
• Understanding of Lambda and Kappa architecture patterns.
• Exposure to Generative AI technologies, tools, and use cases.
• Banking, Financial Services, and Insurance (BFSI) domain experience.
• Bachelor's degree in Computer Science, Information Technology, Engineering, or equivalent educational qualification.
• 12-14+ years of relevant experience in Data Engineering, Big Data, and Cloud technologies.