Lead Databricks Data Engineer

• Job Description: Lead Databricks Data Engineer
• Job Title: Lead Databricks Data Engineer
• Experience: 8-10 Years
• Location: US
• Employment Type: Full-Time

Job Summary

We are looking for a highly experienced Lead Databricks Data Engineer with 8-10 years of experience in Data Engineering, Big Data, and Cloud Data Platforms. The ideal candidate will lead the design, development, modernization, and optimization of enterprise-scale data platforms using Databricks Lakehouse architecture. This role requires strong technical leadership, solution architecture expertise, stakeholder management skills, and hands-on experience building scalable cloud-native data ecosystems.

The candidate will be responsible for defining data engineering standards, mentoring teams, driving modernization initiatives, and delivering high-performance data solutions that support analytics, AI/ML, and business intelligence initiatives.

Key Responsibilities Data Engineering & Architecture

• Design and implement enterprise-scale data platforms using Databricks Lakehouse architecture.
• Architect and develop scalable ETL/ELT data pipelines for batch and real-time processing.
• Define enterprise data models, ingestion frameworks, and transformation strategies.
• Lead the implementation of Delta Lake, Unity Catalog, and data governance frameworks.
• Build and optimize highly scalable Spark and PySpark solutions handling large-volume datasets.
• Drive cloud data platform modernization and migration initiatives.

Technical Leadership

• Serve as the technical lead for Databricks and Data Engineering programs.
• Establish engineering best practices, coding standards, and architecture guidelines.
• Conduct architecture reviews, code reviews, and solution assessments.
• Mentor and guide Data Engineers, Technical Leads, and junior team members.
• Collaborate with Enterprise Architects, Product Owners, Data Scientists, and Business Stakeholders.

Cloud & Platform Engineering

• Design cloud-native data solutions on Azure, AWS, or GCP.
• Implement secure, scalable, and resilient data architectures.
• Optimize costs, performance, reliability, and availability of data platforms.
• Lead CI/CD and Infrastructure-as-Code implementation for data workloads.
• Define and implement monitoring, observability, and operational excellence practices.

Governance & Data Quality

• Establish data governance, lineage, cataloging, and security controls.
• Ensure compliance with organizational and regulatory requirements.
• Implement data quality frameworks and monitoring solutions.
• Collaborate with security and compliance teams to enforce data protection standards.

Stakeholder & Delivery Management

• Partner with business leaders to understand strategic data requirements.
• Translate business requirements into scalable technical solutions.
• Drive delivery planning, risk management, and resource coordination.
• Present architecture decisions and technical recommendations to leadership teams.

Required Technical Skills Databricks

Databricks Lakehouse Platform

Databricks Workflows

Delta Lake

Unity Catalog

Structured Streaming

Databricks SQL

Delta Live Tables (DLT)

Databricks Asset Bundles

Big Data Technologies

Apache Spark

Spark SQL

Kafka

Real-Time Streaming Architectures

Data Lake & Lakehouse Architectures

Cloud Platforms (Strong Experience in One or More) Azure

Azure Databricks

Azure Data Factory (ADF)

ADLS Gen2

Azure Synapse Analytics

Azure Key Vault

Event Hub

AWS

S3

EMR

Glue

Redshift

Lambda

IAM

GCP

BigQuery

Cloud Storage

Dataflow

Pub/Sub

Composer

Programming

Python

SQL

PySpark

Scala (Preferred)

Shell Scripting

DevOps & Automation

Git

Azure DevOps

GitHub Actions

Jenkins

Terraform

Docker

Kubernetes

Data Warehousing

Dimensional Modeling

Star & Snowflake Schema

Data Vault

Master Data Management

Data Governance Frameworks

Required Experience

• 12-15 years of overall IT experience.
• Minimum 6+ years of hands-on Databricks experience.
• 7+ years of experience in Spark/PySpark-based data engineering.
• Experience designing enterprise-scale Data Lake and Lakehouse solutions.
• Strong experience leading large data transformation and modernization programs.
• Experience handling structured, semi-structured, and unstructured data processing.
• Experience supporting analytics, AI/ML, and reporting ecosystems.
• Proven experience managing technical teams and driving architectural decisions.

Preferred Qualifications

• Bachelor's or Master's Degree in Computer Science, Engineering, Information Technology, or related field.
• Databricks Certified Data Engineer Professional.
• Azure Data Engineer Associate.
• AWS Data Analytics Specialty.
• Google Professional Data Engineer Certification.
• TOGAF or Cloud Architecture certifications are a plus.
• Nice to Have
• MLOps and AI Engineering exposure.
• Snowflake and Databricks integration experience.
• Microsoft Fabric exposure.
• Data Mesh implementation experience.
• Data Governance tools such as Collibra, Alation, or Informatica.
• Expertise in Data Quality frameworks such as Great Expectations or Deequ.

Key Competencies

Solution Architecture

Technical Leadership

Stakeholder Management

Program Delivery

Data Strategy & Governance

Team Mentoring & Coaching

Problem Solving & Decision Making

Communication & Executive Presentation Skills

Performance Optimization

Innovation & Continuous Improvement

Technology Classification

• Technology Tower: Data & AI
• Practice Area: Data Engineering
• Primary Skill: Databricks Lakehouse Engineering
• Secondary Skills: Apache Spark, PySpark, Cloud Data Platforms (Azure/AWS/GCP), Data Architecture, Data Governance, MLOps.
• Design and implement data pipelines, Optimize data processing and storage, Ensure data solutions meet performance standards, Provide technical support, Collaborate with stakeholders.
• Bachelor's/Master's in Engineering 5-8 years