Kumaran Systems Technology/Software

Data Engineering Modernization Architect

Chennai, India In person Full-time

12+ years of overall experience in data engineering, data platforms, analytics, or enterprise data architecture, with 5+ years in data platform modernization/cloud data engineering and strong hands-on expertise in Databricks, Microsoft Fabric and Azure Data Factory.

Role

Overview

We are looking for an

  • experienced Data Engineering Modernization Architect to lead the
  • transformation of legacy data platforms, ETL pipelines, and data
  • warehouses into modern, scalable, cloud-native data platforms.
  • The architect will define the target-state
  • data architecture, modernization strategy, migration roadmap, engineering
  • standards, and reusable frameworks. The role requires a strong combination
  • of architecture, hands-on engineering, cloud technologies, data modeling,
  • migration, performance optimization, and automation.
  • The ideal candidate should have
  • successfully delivered at least two enterprise-scale data engineering
  • modernization projects, involving migration from legacy ETL/data warehouse
  • technologies to platforms such as Databricks and Microsoft Fabric, One
  • Lake with Azure Data Factory or equivalent orchestration.

Key Responsibilities

  • Lead modernization of legacy
  • data platforms, ETL/ELT processes and data warehouses into modern
  • cloud-based architectures. Assess existing data platforms, pipelines,
  • databases, ETL jobs, data models and workloads. Define As-Is, To-Be and
  • transition architectures, Develop modernization roadmaps, Identify
  • opportunities to simplify and consolidate legacy data pipelines, Define
  • reusable modernization patterns and frameworks.

Key Skills Requirement

  • Strong hands-on and
  • architectural expertise in Databricks Lakehouse architecture, PySpark,
  • Delta Lake, Delta Live Tables / Lakeflow capabilities, Unity Catalog,
  • Workflows / Jobs, Databricks SQL, Change Data Capture, Schema evolution,
  • Databricks security and governance, Azure Data Factory ADF pipelines, Copy
  • activities, Mapping Data Flows, Parameters and variables, Metadata-driven
  • pipelines, Dynamic pipelines, Integration Runtime, Self-hosted Integration
  • Runtime, Trigger mechanisms, Error handling, Retry frameworks, Dependency
  • management, Pipeline monitoring, CI/CD
  • The candidate should be strong
  • in Enterprise Data Architecture, Modern Data Lakehouse, Data Warehouse,
  • Data Lake, Lakehouse architecture, Medallion architecture, Dimensional
  • modelling, Star/Snowflake schemas, Data Vault, Data mesh concepts, Batch
  • and streaming architecture, CDC architecture, Event-driven data pipelines,
  • Master/reference data, Data governance, Data lineage, Data quality,
  • Metadata management
  • Strong experience in developing
  • reusable frameworks for Metadata-driven ingestion, Dynamic pipeline
  • generation, Automated schema discovery, Automated table creation,
  • Incremental load framework, CDC framework, Data quality framework,
  • Pipeline monitoring, Exception handling, Automated reconciliation, Data
  • validation, Deployment automation, Environment provisioning
  • Strong experience in Legacy ETL
  • modernization, Data warehouse migration, Data lake migration, ETL → ELT
  • transformation, On-premises → Azure migration, Legacy database → Lakehouse
  • migration, Batch → near-real-time modernization, Stored procedure
  • modernization, Legacy ETL code conversion.

Job description provided by Kumaran Systems.
View the original posting