- Job title : Senior Data Engineer
- Exp : 5-8 years exp
- Azure Databricks: Strong hands-on experience with Databricks Lakehouse
- Architecture, Delta Lake, Unity Catalog, Databricks Apps, Workflows/Jobs, and
- Databricks Asset Bundles (DAB).
- Real-Time Data Ingestion: Experience with Azure Event Hubs, Spark Structured
- Streaming, Auto Loader, Databricks Zerobus Ingest, and Delta Change Data Feed
- (CDF).
- Python & SQL: Strong proficiency in Python, PySpark, and SQL for
- data engineering, transformation, optimization, and automation.
- Configuration-Driven Framework: Experience designing reusable, modular,
- metadata-driven Python frameworks with configurable business rules, execution
- engines, and orchestration layers.
- Lakebase (PostgreSQL): Hands-on experience with Lakebase for operational
- data storage, relational data modeling, SQL development, and Lakebase-to-Delta
- synchronization.
- Databricks Lakehouse: Experience implementing Bronze, Silver, and Gold
- architectures, Delta tables, data transformations, and scalable analytical
- pipelines.
- Data Pipeline Engineering: Ability to design batch and streaming
- pipelines with error handling, retries, replay, reconciliation, and failure
- recovery mechanisms.
- Data Quality & Governance: Experience implementing validation
- frameworks, data quality checks, quarantine handling, audit logging, and Unity
- Catalog governance.
- Data Modeling & Historical Processing: Strong understanding of
- relational data modeling, source-to-target mappings, CDC, SCD Type 2, and
- historical data synchronization.
- Performance Optimization: Ability to optimize Spark workloads, SQL
- queries, Delta tables, job execution, and data processing costs.
Job description provided by Elfonze Technologies.
View the original posting
