Lead DevOps/Platform Engineer IV

Overview

At PNNL, our core capabilities are divided among major departments that we refer to as Directorates within the Lab, focused on a specific area of scientific research or other function, with its own leadership team and dedicated budget.

Our Science & Technology directorates include National Security, Earth and Biological Sciences, Physical and Computational Sciences, and Energy and Environment. In addition, we have an Environmental Molecular Sciences Laboratory, a Department of Energy, Office of Science user facility housed on the PNNL campus.

The National Security Directorate (NSD) drives science-based, mission-focused solutions to take on complex, real-world threats to our nation and the world.

The AI and Data Analytics Division, part of NSD, combines profound domain expertise and creative integration of advanced hardware and software to deliver computational solutions that address complex data and analytic challenges. Working in multidisciplinary teams, we connect foundational research to engineering to operations, providing the tools to innovate quickly and field results faster. Our strengths are integrated across the data analytics lifecycle, from data acquisition and management to analysis and decision support.

Responsibilities

We are seeking a Lead DevOps/Platform Engineer to join PNNL's advanced AI engineering initiatives, contributing to next-generation systems spanning agentic AI platforms, large-scale data orchestration, and real-time intelligence processing. In this role, you'll apply your expertise in scalable system design and AI/ML engineering to build mission-critical capabilities while developing your technical leadership and establishing yourself as a key contributor to our engineering community.

Who You Are

You're an accomplished engineer with strong foundations in DevOps, scalable system design, AI/ML development, and production software engineering. You're ready to take on increasing technical responsibility, leading components of complex systems while mentoring junior team members. You excel at translating technical requirements into working solutions, selecting appropriate approaches for challenging problems, and contributing meaningfully to technical direction and project success.

What You'll Build

AI-Native Systems & Platforms

• Design and deploy scalable agentic AI systems with dynamic reasoning and decision-making capabilities
• Architect LLM orchestration frameworks using LangChain, LlamaIndex, and emerging agent platforms
• Build MLOps platforms spanning experiment tracking, model versioning, deployment, and governance
• Develop developer-focused tooling, adapters, and interfaces for AI-native frameworks
• Integrate multi-modal data sources (text, vision, structured/sensor data) into cohesive reasoning pipelines

Scalable Infrastructure & Data Systems

• Design microservices architectures coordinating across multiple domains and security enclaves
• Lead distributed system design processing data from hundreds of sources simultaneously
• Architect real-time streaming platforms handling terabytes per hour with event-driven architectures
• Build robust data pipelines for petabyte-scale ETL, data lake/mesh architectures, and real-time analytics
• Design container orchestration (Kubernetes) and CI/CD pipelines for classified and edge environments

Mission-Critical Production Systems

• Deploy AI systems in highly secure environments with resilient agent-to-agent communications
• Create monitoring and observability systems (logging, metrics, tracing) across secure enclaves
• Ensure compliance with ethical AI standards and security-first DevOps practices
• Build geospatial processing, time-series, and intelligence data fusion capabilities

Technical Leadership

• Lead a team of engineers to deliver on high risk / high impact ambiguous technical scope
• Drive technical strategy and architectural decisions across cross-functional teams
• Translate ambiguous requirements and cutting-edge research into actionable technical roadmaps
• Lead design discussions shaping team-wide engineering standards
• Mentor engineering teams and guide junior scientists/engineers

Technical Knowledge, Skills, and Abilities

Platform Architecture & Infrastructure Leadership

• Expert-level proficiency in Python and at least one additional language (Go, C#/.NET, C++) with proven ability to establish infrastructure automation standards, architect scalable tooling platforms, and guide teams in developing sophisticated automation frameworks
• Mastery of Infrastructure as Code principles with deep expertise in Terraform, CloudFormation, Pulumi, or ARM templates and demonstrated ability to design enterprise-wide IaC strategies, module libraries, and governance frameworks that enable consistent and secure infrastructure deployment
• Proven track record of architecting and leading implementation of enterprise-grade CI/CD platforms with ability to define build/release strategies, establish deployment patterns, and drive continuous delivery adoption while designing internal developer platforms that abstract complexity and accelerate team velocity
• Expert proficiency with GitOps methodologies (ArgoCD, Flux), infrastructure testing frameworks (Terratest, InSpec), and policy-as-code (OPA, Sentinel) with strategic application of AI assist tools to drive team productivity, accelerate automation development, and optimize operational efficiency

Cloud Architecture & Orchestration Expertise

• Demonstrated expertise architecting and leading multi-cloud infrastructure strategies across AWS, Azure, and GCP with deep expertise in containerization and Kubernetes ecosystem including production-grade container platforms, custom operators, CRDs, and multi-cluster strategies at organizational scale
• Expert ability to architect sophisticated event-driven systems using cloud-native services (EventBridge, Event Grid, Pub/Sub, SNS/SQS) with advanced knowledge of service mesh architectures (Istio, Linkerd, Consul) and API gateway patterns for zero-trust networking and complex microservice environments
• Mastery of cloud and container networking including CNI design, custom ingress implementations, advanced load balancing, service discovery patterns, and network security policies with ability to troubleshoot complex distributed system networking issues
• Experience architecting edge computing solutions, hybrid cloud strategies, and secure enclave deployments with understanding of data sovereignty, latency optimization, and security requirements for geographically distributed infrastructure

Reliability Engineering & Security Leadership

• Proven ability to architect comprehensive observability platforms integrating metrics (Prometheus, Thanos, Cortex), distributed tracing (Jaeger, Tempo), and logging systems (ELK, Loki, Splunk) with deep expertise in SRE principles including SLO/SLI frameworks, error budgets, and incident management
• Expert implementation of security-first infrastructure including secrets management (Vault, AWS Secrets Manager, Azure Key Vault), automated vulnerability scanning, DevSecOps toolchains, and security policy enforcement across all infrastructure layers
• Strategic capability to design enterprise disaster recovery and business continuity strategies including multi-region architectures, automated backup systems, RPO/RTO optimization, and regular DR testing with advanced chaos engineering practices to systematically improve system resilience
• Deep understanding of compliance frameworks (SOC 2, HIPAA, FedRAMP, PCI-DSS, GDPR) with proven ability to implement automated compliance controls, audit logging, and infrastructure hardening standards that meet regulatory requirements

MLOps & Data Platform Engineering

• Expertise in architecting end-to-end MLOps platforms with proven ability to design and implement model lifecycle management infrastructure including experiment tracking (MLflow, Weights & Biases), model versioning, model registries, feature stores (Feast, Tecton), and automated ML pipeline orchestration supporting continuous training and deployment
• Deep expertise in building infrastructure for ML model serving and deployment including real-time inference APIs, batch prediction systems, A/B testing frameworks, model monitoring for drift detection, and automated model retraining pipelines with canary deployments and rollback capabilities
• Advanced knowledge of distributed ML training infrastructure including multi-GPU and multi-node training orchestration, resource scheduling, and optimization for frameworks like PyTorch, TensorFlow, and JAX on Kubernetes-based platforms (Kubeflow, Ray, Spark ML) with deep understanding of compute resource management and cost optimization
• Proven ability to architect cloud-native data platforms with expertise in ETL/ELT orchestration frameworks (Airflow, Prefect, Dagster, AWS Step Functions), production data storage systems (S3, Redshift, Databricks Delta Lake, PostgreSQL, MongoDB, Snowflake), and distributed data processing frameworks (Spark/Databricks, Kafka, Flink, Ray) supporting petabyte-scale data systems and real-time ML feature pipelines

Technical Leadership & Strategic Impact

• Exceptional problem-solving and troubleshooting abilities with proven track record of resolving complex infrastructure incidents spanning ML pipelines, data platforms, and distributed systems while leading incident response and root cause analysis, combined with outstanding communication skills to translate technical complexity into business impact for executive leadership and stakeholders
• Demonstrated ability to establish infrastructure and MLOps documentation standards, create comprehensive runbooks for ML system operations and DR procedures, develop technical training programs, and build knowledge sharing practices while mentoring and developing platform engineering teams through technical guidance and architecture reviews
• Proven capacity to lead multiple concurrent infras