Design and develop systems or enhancements to existing telemetry, monitoring, and logging infrastructure to improve observability and debugging for Dataproc and Managed Service for Apache Spark. Architect and optimize observability pipelines for Spark Streaming and Structured Streaming workloads, providing insights into micro-batch latencies, checkpointing bottlenecks, and input message lag. Apply Apache Spark core internals knowledge to build tools that detect performance regressions, data skew, and resource inefficiencies in customer workloads.
Collaborate with cross-functional partners (e.g., Product, UX) to design and deliver intuitive monitoring dashboards, cost-attribution features, and developer tools. Collaborate with Dataproc and Cloud Platform teams to integrate and release telemetry features, OpenTelemetry (OTel) pipelines, and diagnostic frameworks safely at scale. Build robust monitoring systems providing visibility, execution history, and real-time performance insights into distributed big data workloads.
Job description provided by Google.
View the original posting
