Senior Site Reliability Engineer, AI/ML

Job Description

We are seeking a highly skilled Senior Site Reliability Engineer to join our Technical Operations team and lead reliability, scalability, and performance initiatives for AI/ML workloads across multi-cloud and on-prem environments. This role will focus on building and maintaining resilient infrastructure for advanced data science workflows, including NVIDIA DGX systems, leveraging platforms such as Domino Data Lab, Slurm, and NVIDIA Base Command, while driving automation, observability, and networking optimization

Key Responsibilities

• Contribute to deployment, and maintenance of infrastructure across AWS, GCP, and Azure, as well as on-prem NVIDIA DGX systems.
• Implement and manage Infrastructure as Code (IaC) using Terraform and Ansible for automated provisioning and configuration.
• Support cloud and on-prem networking solutions for secure, high-performance connectivity.
• Manage and optimize Domino Data Lab workflows and Slurm clusters for distributed training and inference.
• Integrate and support NVIDIA Base Command for GPU-based compute environments.
• Develop automation scripts and tools in Python to streamline operations and improve reliability.
• Support CI/CD pipelines using GitLab, ensuring smooth deployments to UAT and production environments.
• Implement and maintain observability solutions (monitoring, logging, alerting) using tools like Prometheus, Grafana, and cloud-native services.
• Deploy and manage Kubernetes clusters (EKS, GKE) for scalable containerized workloads.
• Troubleshoot complex workflows and ensure high availability of critical systems.
• Collaborate with data science and engineering teams to optimize resource utilization and workflow efficiency.
• Drive best practices for incident response, capacity planning, and system reliability in multi-cloud and HPC environments.

Additional Responsibilities

• Administer and optimize ITSM platforms (e.g., Jira Service Management, ServiceNow) for release/change/incident workflows.
• Support tooling across CI/CD, monitoring, and ticketing systems to ensure traceability and automation.
• Maintain documentation and evidence for audits related to release/change/incident processes.
• Partner with Compliance and InfoSec teams to ensure controls meet HIPAA, HITRUST, FDA GxP, and ISO 27001 standards.
• Act as the primary liaison between engineering, product, support, and compliance teams for operational readiness.
• Facilitate regular status updates, incident reviews, RCA’s and change planning sessions with stakeholders.
• Support in updating onboarding materials and training sessions for engineers and product managers on release/change/incident protocols.
• Promote a culture of ownership and reliability through education and process transparency.
• Support retrospectives for major releases and incidents to identify process gaps and improvement opportunities.
• Track and report on KPIs such as change success rate, incident recurrence, and release velocity.
• Identify operational risks and escalate proactively to leadership.
• Maintain escalation matrices and ensure readiness for high-severity incidents.