Core Technical Skills
Strong Python programming experience
Advanced SQL
DBT development
Google BigQuery
PySpark
- Data engineering and ETL/ELT pipeline development
- Data validation and reconciliation
- Data migration experience
Python Libraries & Frameworks
Pandas
NumPy
PyArrow
Pytest
Great Expectations or similar data quality framework
Google Cloud Python SDK
GCP Services
IAM
Cloud Logging and Monitoring
Lead smallscale data engineering projects. Ensure stable data processing operations. Mentor junior engineers. Provide project support.
SAS Migration & Code Conversion
- Analyse existing SAS datasets, SAS programs, PROC SQL, Data Steps, and SAS macros.
- Convert SAS business logic into Python, SQL, PySpark, and DBT models.
- Support migration of historical and incremental SAS data into GCP.
- Document SAS-to-GCP transformation logic and technical conversion rules.
- Assist in identifying reusable patterns for SAS macro conversion.
Python Data Engineering
- Develop Python-based data processing, automation, and validation scripts.
- Build reusable Python frameworks for data ingestion, transformation, and reconciliation.
- Use Python libraries such as Pandas, NumPy, PyArrow, and Google Cloud SDK.
- Automate data quality checks, file processing, metadata extraction, and exception reporting.
- Develop utilities to compare SAS outputs with GCP outputs.
DBT & BigQuery Development
- Develop and maintain DBT models for transformation logic.
- Convert SAS PROC SQL logic into DBT SQL models.
- Implement DBT tests for data quality, uniqueness, referential integrity, and business rules.
- Support Bronze, Silver, and Gold layer implementation in BigQuery.
- Optimize SQL queries and BigQuery tables for performance and cost efficiency.
GCP Data Pipeline Development
Build scalable data pipelines using GCP services such as
BigQuery
Cloud Storage
Cloud Composer / Airflow
Dataproc
Dataflow
Cloud Functions
- Develop batch and incremental data processing workflows.
- Support orchestration, monitoring, and production deployment of pipelines.
- Implement secure data movement from legacy SAS platforms to GCP.
Data Quality, Validation & Reconciliation
- Perform source-to-target reconciliation between SAS and GCP.
- Validate record counts, financial totals, business rules, and derived metrics.
- Investigate and resolve data mismatches and transformation defects.
- Build automated reconciliation reports using Python and SQL.
- Support SIT, UAT, regression testing, and production validation.
Job description provided by EXL Service.
View the original posting
