- Develop API-driven systems to govern, manage, and monitor large-scale batch Big Data applications.
- Build scalable backend services and data engineering solutions that support data processing and operational workflows.
- Develop and maintain data transformation processes using Spark, SQL, Hive, Python, Scala, and related technologies.
- Work with AWS cloud services such as S3, EC2, EMR, Lambda, DynamoDB, and API Gateway to build cloud-native data and backend solutions.
- Build and enhance workflow orchestration using Apache Airflow, including advanced DAG design, dependency management, scheduling, monitoring, and failure handling.
- Define technical scope, objectives, and implementation approaches by participating in requirements gathering, technical research, and process definition.
- Participate in architecture reviews, code reviews, performance tuning, and operational readiness activities.
- Contribute to test planning and validation for application integrations, functional areas, and project deliverables.
- Partner with product owners, data engineers, backend engineers, QA, DevOps, and other cross-functional teams to deliver high-quality solutions in an Agile/Scrum environment.
- Troubleshoot production issues, improve system reliability, and drive automation across data and backend workflows.
- 5+ years of hands-on experience developing enterprise-scale applications, data platforms, or distributed systems.
- 5+ years of experience developing, and operating Big Data platforms and cloud-based infrastructure services, preferably using AWS EMR and the Hadoop ecosystem.
- Hands-on experience with AWS, Spark, Python and/or Scala, Airflow, SQL, Hive, and related data processing technologies.
- Proficiency in either Python or Scala, with the ability to build production-grade data pipelines and backend services.
- Experience with AWS cloud services, including S3, EC2, EMR, Lambda, DynamoDB, and API Gateway.
- Strong experience with Apache Airflow orchestration, including complex workflows, scheduling, dependency handling, monitoring, and operational support.
- Strong understanding of data engineering concepts, including data pipelines, data transformation, batch processing, real-time processing, data quality, and performance optimization.
- Excellent analytical and problem-solving skills.
- Strong communication and presentation skills, with the ability to explain technical concepts clearly to both technical and non-technical audiences.
- Experience with version control and CI/CD tools, including Git and Jenkins, for source code management, build automation, testing, and deployment workflows.
- Ability to work effectively in cross-functional Agile/Scrum teams.
- Experience with LLMs, Generative AI, Agentic AI, or AI-assisted engineering workflows.
- Experience with API design, microservices, event-driven architecture, or serverless backend development.
- Experience with CI/CD pipelines, infrastructure as code, automated testing, and production deployment practices.
- Experience with data governance, metadata management, lineage, data observability, or operational controls for enterprise data platforms
Job description provided by Bristlecone.
View the original posting