Observability Engineer

Location: Westlake

Position Overview

The Observability Engineer builds and improves the capabilities used to understand the health, performance, and reliability of Goosehead applications and services. This hands-on role focuses primarily on application and service observability and partners with Cloud & Infrastructure on underlying infrastructure telemetry.

The engineer works with Software Engineering, DevOps, NOC, Incident Management, and Production Support to instrument services, improve telemetry, establish service-health and alerting standards, and accelerate production diagnosis. New Relic is a key platform, but the role is broader than administration of any single tool.

Core outcome: Make production health visible, alerts actionable, and failures easier to diagnose.

Key Responsibilities

• Design, implement, and improve observability for production applications, APIs, integrations, and critical services using OpenTelemetry for instrumenting metrics, logs, traces, and custom telemetry.
• Partner with Engineering to instrument applications using OpenTelemetry for structured logging, distributed tracing, and custom metrics.
• Create and maintain service-health dashboards, alerts, service maps, and diagnostic views that make availability, latency, errors, dependencies, and customer impact clear.
• Establish practical standards for instrumentation, monitoring coverage, alerting, dashboards, Service Level Indicators (SLIs), and Service Level Objectives (SLOs).
• Improve alert quality by reducing noise, false positives, duplicates, and non-actionable alerts; ensure alerts support effective NOC triage and escalation.
• Support Incident Management and Production Support during significant events by correlating telemetry, deployments, dependencies, and system behavior to speed diagnosis.
• Partner with Problem Management to identify observability gaps after recurring or significant incidents and implement preventive improvements.
• Partner with DevOps and Cloud & Infrastructure to connect application observability with CI/CD, deployments, cloud, infrastructure, and platform telemetry.
• Build scripts and automation using PowerShell, Python, Bash, APIs, or similar technologies to improve telemetry, configuration, alerting, and operational workflows.
• Help integrate observability and alert/event data with ITSM and incident workflows, including future ServiceNow capabilities.
• Maintain concise technical standards, reusable patterns, and operational guidance for Engineering and Technology Operations teams.
• Maintain service-based error budgets which our development teams use to dictate feature work.
• Minimum 3 years of hands-on experience in Observability Engineering, SRE, DevOps, Application Performance Monitoring, Production Engineering, Application Support Engineering, or a closely related role; 4-5 years preferred.
• Hands-on experience with application performance monitoring, metrics, centralized logging, distributed tracing, dashboards, alerting, and production troubleshooting.
• Experience instrumenting applications or services using OpenTelemetry for custom metrics, structured logs, and distributed tracing.
• Strong understanding of APIs, integrations, distributed applications, service dependencies, and common performance and reliability issues.
• Ability to correlate metrics, logs, traces, and change/deployment information to troubleshoot production issues.
• Hands-on scripting or automation experience using PowerShell, Python, Bash, APIs, or comparable technologies.
• Working knowledge of SLIs, SLOs, availability, latency, error rates, alert design, and service-health concepts.
• Working knowledge of cloud environments and the relationship between application and infrastructure telemetry.
• Strong troubleshooting, analytical, communication, and cross-team collaboration skills.
• Associate or bachelor's degree in Information Technology, Computer Science, Engineering, or related field preferred; equivalent technical experience considered.

Preferred Qualifications

• Hands-on experience with New Relic, including OpenTelemetry services, logs, dashboards, alerting, distributed tracing, or service maps.
• Experience with comparable observability platforms such as Datadog, Dynatrace, Splunk Observability, AppDynamics, Azure Monitor/Application Insights, Grafana, or Prometheus.
• Practical OpenTelemetry implementation experience and experience defining reusable observability or instrumentation standards.
• Experience establishing SLIs/SLOs or service-health measures for production applications.
• Experience integrating observability platforms with incident, event, notification, or ITSM workflows; ServiceNow experience is a plus.
• Experience supporting Microsoft Azure or other public cloud environments and integrating observability with CI/CD or deployment processes.
• Familiarity with ITIL Incident Management and Problem Management practices.