Production Systems Engineer, Fleet AI Systems

Meta is seeking a Production Systems Engineer to lead the reliability, scalability, and operational excellence of our AI infrastructure fleet. This role sits at the intersection of large-scale AI compute systems and production engineering, supporting the hardware and software stack that powers Meta's AI training and inference workloads across tens of thousands of accelerators and servers. You will drive the strategy for fleet health, automated diagnostics, and lifecycle management of AI-optimized hardware, partnering closely with hardware design, silicon, software, and data center operations teams to ensure our AI systems are production-ready and operating at peak efficiency.

Responsibilities

• Define and drive the long-term technical strategy for fleet health monitoring, automated remediation, and lifecycle management of AI compute infrastructure at hyperscale
• Lead cross-functional initiatives spanning hardware design, firmware, systems software, and data center operations to bring new AI server platforms from pre-production validation through mass deployment
• Architect and oversee development of large-scale test automation frameworks for AI accelerator systems, covering GPU, TPU, and custom silicon configurations across diverse fleet topologies
• Establish organizational standards for root cause analysis methodologies, failure mode classification, and escalation workflows for AI system incidents at fleet scale
• Drive vendor and partner engagement on hardware specifications, firmware quality, and test methodology improvements to continuously raise the reliability bar for AI infrastructure
• Develop and champion data-driven visibility platforms that surface fleet health signals, predictive failure indicators, and capacity risk across AI training and inference clusters
• Lead ambiguous, high-impact investigations into systemic hardware and software failures affecting AI workload availability, producing durable solutions and documented learnings
• Mentor and provide technical direction to other engineers across the production systems organization, shaping engineering practices and raising the collective capability of the team
• Collaborate with capacity engineering and AI platform teams to translate fleet reliability requirements into hardware procurement, qualification, and deployment criteria
• Influence the roadmap of internal tooling and automation systems to reduce mean time to detection and resolution for AI fleet incidents at scale

Minimum qualifications

• Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience
• 12+ years of experience in production systems engineering, server hardware support, or infrastructure reliability engineering at hyperscale
• 12+ years of experience troubleshooting and root-causing failures across the full hardware and software stack, including CPUs, accelerators, memory, networking, storage, firmware, and operating systems
• Experience leading large-scale technical programs across hardware, firmware, and software engineering organizations with demonstrated organizational impact
• Experience designing and implementing automated diagnostics, test frameworks, or remediation systems for large server fleets
• Experience with Linux systems administration and scripting or programming in Python or equivalent languages for infrastructure tooling

Preferred qualifications

• Experience supporting AI training or inference infrastructure, including GPU clusters, high-speed interconnects such as InfiniBand or RoCE, and AI accelerator hardware at fleet scale
• Experience engaging directly with silicon vendors or ODMs on firmware qualification, failure analysis, and hardware specification development for AI compute systems
• Experience in post-production hyperscale environments managing full hardware lifecycle progressions for AI or HPC server platforms
• Experience with predictive fleet health modeling, telemetry pipelines, or data visualization platforms applied to large-scale hardware operations