12+ years of overall experience in data engineering, data platforms, analytics, or enterprise data architecture, with 5+ years in data platform modernization/cloud data engineering and strong hands-on expertise in Databricks, Microsoft Fabric and Azure Data Factory.
Role
Overview
We are looking for an
- experienced Data Engineering Modernization Architect to lead the
- transformation of legacy data platforms, ETL pipelines, and data
- warehouses into modern, scalable, cloud-native data platforms.
- The architect will define the target-state
- data architecture, modernization strategy, migration roadmap, engineering
- standards, and reusable frameworks. The role requires a strong combination
- of architecture, hands-on engineering, cloud technologies, data modeling,
- migration, performance optimization, and automation.
- The ideal candidate should have
- successfully delivered at least two enterprise-scale data engineering
- modernization projects, involving migration from legacy ETL/data warehouse
- technologies to platforms such as Databricks and Microsoft Fabric, One
- Lake with Azure Data Factory or equivalent orchestration.
Key Responsibilities
- Lead modernization of legacy
- data platforms, ETL/ELT processes and data warehouses into modern
- cloud-based architectures. Assess existing data platforms, pipelines,
- databases, ETL jobs, data models and workloads. Define As-Is, To-Be and
- transition architectures, Develop modernization roadmaps, Identify
- opportunities to simplify and consolidate legacy data pipelines, Define
- reusable modernization patterns and frameworks.
Key Skills Requirement
- Strong hands-on and
- architectural expertise in Databricks Lakehouse architecture, PySpark,
- Delta Lake, Delta Live Tables / Lakeflow capabilities, Unity Catalog,
- Workflows / Jobs, Databricks SQL, Change Data Capture, Schema evolution,
- Databricks security and governance, Azure Data Factory ADF pipelines, Copy
- activities, Mapping Data Flows, Parameters and variables, Metadata-driven
- pipelines, Dynamic pipelines, Integration Runtime, Self-hosted Integration
- Runtime, Trigger mechanisms, Error handling, Retry frameworks, Dependency
- management, Pipeline monitoring, CI/CD
- The candidate should be strong
- in Enterprise Data Architecture, Modern Data Lakehouse, Data Warehouse,
- Data Lake, Lakehouse architecture, Medallion architecture, Dimensional
- modelling, Star/Snowflake schemas, Data Vault, Data mesh concepts, Batch
- and streaming architecture, CDC architecture, Event-driven data pipelines,
- Master/reference data, Data governance, Data lineage, Data quality,
- Metadata management
- Strong experience in developing
- reusable frameworks for Metadata-driven ingestion, Dynamic pipeline
- generation, Automated schema discovery, Automated table creation,
- Incremental load framework, CDC framework, Data quality framework,
- Pipeline monitoring, Exception handling, Automated reconciliation, Data
- validation, Deployment automation, Environment provisioning
- Strong experience in Legacy ETL
- modernization, Data warehouse migration, Data lake migration, ETL → ELT
- transformation, On-premises → Azure migration, Legacy database → Lakehouse
- migration, Batch → near-real-time modernization, Stored procedure
- modernization, Legacy ETL code conversion.
Job description provided by Kumaran Systems.
View the original posting