Core Skills & Experience
- Strong hands-on experience designing, building, and optimizing enterprise-scale data pipelines, data architecture, and data products for performance, scalability, and efficient AI agent consumption.
- Deep expertise in Starburst/Trino and distributed SQL, including query optimization, materialized views (MVs), data modeling, storage design, and tuning data products for low-latency, high-concurrency AI agent access.
- Strong experience optimizing data storage, partitioning, indexing, file formats, and materialization strategies to improve query performance, data freshness, and cost efficiency.
- Hands-on development experience with Apache Spark using Scala and/or Python/PySpark, including migrating complex, SQL-intensive transformations to distributed processing frameworks to improve data curation speed and efficiency.
- Strong ability to evaluate and redesign data processing workflows, selecting the appropriate technologies and execution patterns across SQL, Spark, and other processing frameworks to optimize performance.
- Experience with RDBMS technologies and modern lakehouse architecture, preferably Apache Iceberg, including table design, partitioning, compaction, and data lifecycle optimization.
- Understanding of integration patterns across NiFi, Kafka, Spark, RDBMS, and lakehouse platforms for efficient batch and real-time data processing.
- Strong software engineering practices, including CI/CD, automated testing, source control, infrastructure automation, observability, and production support.
- Experience with data governance, metadata management, lineage, data quality, and secure data access in enterprise environments.
NA
NA
Job description provided by Zensar.
View the original posting
