Data Engineer

• Title: Senior Technical Specialist - PBI Databricks
• Location: Remote
• Duration: 12+ Months
• Education: Any Degree
• Exp Level: 8+ Years

Job Responsibilities

Design and build conformed dimensions and transactional fact tables for enterprise reporting.

Develop and maintain data pipelines on Azure Databricks using PySpark, Spark SQL, and Delta Lake.

Implement best practices in dimensional modeling, including

• Star schemas
• Conformed dimensions
• Surrogate and deterministic hash keys (e.g., SHA-256)
• Transactional fact design
• Translate existing SQL Server and stored procedure logic into efficient Spark-based transformations.
• Implement incremental data processing and change data capture (CDC) pipelines.
• Ensure data reconciliation with source systems to meet accuracy and audit standards.
• Collaborate closely with a senior Databricks lead to deliver core data assets within defined timelines.

Required Skills

8+ years of experience in data engineering, with strong focus on Azure Databricks.

Proficiency in

PySpark

Spark SQL

Delta Lake

Deep expertise in dimensional modelling and data warehousing concepts.

Experience with

• Building scalable production pipelines
• Incremental data loads and CDC patterns
• Performance optimization in distributed systems
• Strong understanding of relational databases and ability to interpret complex SQL/stored procedures
• Good to have

Experience with

Lakeflow / Spark Declarative Pipelines or dbt

Unity Catalog

Exposure to financial/General Ledger data models

Familiarity with JD Edwards systems, including

F0902 (balances)

F0911 (journal details)