Data Platform Engineer

Job Summary: 2-3 sentence explanation of this role’s objectives and impact Administers BEP's Databricks platform and serves as the resident Databricks expert and data engineer. Owns platform permissions, security, and governance, and supports and improves existing data infrastructure across Databricks and AWS. Works with BI, Data Science, AI, DevOps, and Security teams to deliver secure, high-performance data. Responsibilities: day-to-day tasks and measurable outcomes Administer the Databricks platform, including workspaces, compute , user permissions, security policies, and usage and cost monitoring.

Support and improve existing data infrastructure, and create and evolve solutions using Databricks and AWS services including S3, Glue, EMR, Redshift, Aurora, and DynamoDB. Lead migration of existing data workloads onto Databricks. Serve as the resident Databricks expert: establish standards and best practices, train teams, and assist developers and analysts with data performance improvements. Monitor Databricks costs and offer solutions for cost optimization Collaborate with BI, Data Science, and AI teams to create high-performance data sets.

Collaborate with DevOps and Security teams to create secure and re-deployable resources. Respond in a timely manner to service issues and requests; occasional after-hours work. Test and propose new technology to solve business problems. Other related duties as assigned. Existing Skills/Abilities: what is required for success in the role Experience administering, developing, and maintaining solutions in Databricks with a focus on performance, cost management, data integration and federation, Unity Catalog design, and data governance/security.

Deep experience building data integration/ETL solutions on at least one distributed computing platform; PySpark is a plus. Solid understanding of data modeling and performance tuning for data warehouses (Redshift a plus ), RDBMS (MySQL, Aurora, or Postgres; MSSQL a plus), and NoSQL (DynamoDB a plus). Thorough understanding and implementation experience of Data Lakehouse Architecture, open table formats including Apache Iceberg, Glue Catalog, and S3-based storage.

Understanding of at least one programming language such as Python, Scala, Golang, NodeJS, or TypeScript, and a high-level understanding of messaging queue and data streaming technology for near/real-time integration. Basic understanding of infrastructure-as-code tools such as Terraform or CloudFormation; Git version control; experience with monitoring, APM, and logging tools (Datadog a plus). Effective written and oral communication skills.

Experience and Education: years of total experience, education level or focus BA/BS in Computer Science, Information Systems, or a related field; or equivalent professional experience. 3+ years of data engineering experience, including 2+ years of hands-on Databricks experience. Experience administering a production Databricks environment, including Unity Catalog, user permissions, and cost management. Experience building and supporting data solutions on AWS in a production environment.

Experience migrating data workloads to a new platform. Experience supporting BI, analytics, or data science teams. Databricks certification (Data Engineer or Platform Administrator) is a plus.