Senior DevOps/Platform Engineer III

Overview

At PNNL, our core capabilities are divided among major departments that we refer to as Directorates within the Lab, focused on a specific area of scientific research or other function, with its own leadership team and dedicated budget.

Our Science & Technology directorates include National Security, Earth and Biological Sciences, Physical and Computational Sciences, and Energy and Environment. In addition, we have an Environmental Molecular Sciences Laboratory, a Department of Energy, Office of Science user facility housed on the PNNL campus.

The National Security Directorate (NSD) drives science-based, mission-focused solutions to take on complex, real-world threats to our nation and the world.

The AI and Data Analytics Division, part of NSD, combines profound domain expertise and creative integration of advanced hardware and software to deliver computational solutions that address complex data and analytic challenges. Working in multidisciplinary teams, we connect foundational research to engineering to operations, providing the tools to innovate quickly and field results faster. Our strengths are integrated across the data analytics lifecycle, from data acquisition and management to analysis and decision support.

Responsibilities

We are seeking a Senior DevOps/Platform Engineer to join PNNL's advanced AI engineering initiatives, contributing to next-generation systems spanning agentic AI platforms, large-scale data orchestration, and real-time intelligence processing. In this role, you'll apply your expertise in scalable system design and AI/ML engineering to build mission-critical capabilities while developing your technical leadership and establishing yourself as a key contributor to our engineering community.

Who You Are

You're an accomplished engineer with strong foundations in DevOps, scalable system design, AI/ML development, and production software engineering. You're ready to take on increasing technical responsibility, leading components of complex systems while mentoring junior team members. You excel at translating technical requirements into working solutions, selecting appropriate approaches for challenging problems, and contributing meaningfully to technical direction and project success.

What You'll Build

AI Systems & Platforms

• Develop and deploy agentic AI systems with reasoning and decision-making capabilities
• Build components of LLM orchestration frameworks using LangChain, LlamaIndex, and emerging platforms
• Contribute to MLOps platforms including experiment tracking, model versioning, and deployment pipelines
• Create developer tooling, utilities, and interfaces for AI-native frameworks
• Integrate multi-modal data sources into cohesive processing pipelines

Scalable Infrastructure & Data Systems

• Develop microservices within distributed architectures handling high-throughput workloads
• Build components of real-time streaming platforms and event-driven systems
• Implement data pipelines for large-scale ETL, data processing, and analytics
• Deploy containerized applications using Kubernetes and support CI/CD pipelines
• Contribute to systems deployed in secure and edge environments

Mission-Critical Production Systems

• Deploy AI systems with appropriate monitoring, logging, and observability
• Ensure code quality, security best practices, and compliance standards
• Build geospatial processing, time-series, and data fusion capabilities
• Support system performance optimization and troubleshooting

Technical Leadership

• Lead technical components of projects and tasks
• Mentor junior staff and contribute to team knowledge sharing
• Participate in design discussions and contribute to architectural decisions
• Support proposal development with technical content and scoping
• Build effective collaborations across teams and S&E domains

Technical Knowledge, Skills, and Abilities

Infrastructure Automation & Platform Engineering

• Demonstrated proficiency in Python and working knowledge of at least one additional language (C#/.NET, Go, C++) for infrastructure automation and tooling development
• Knowledge of Infrastructure as Code principles and tools including Terraform, CloudFormation, Pulumi, or ARM templates with emphasis on modular, reusable code patterns
• Ability to design, implement, and maintain sophisticated CI/CD pipelines across multiple environments using tools such as Jenkins, GitLab CI, GitHub Actions, or Azure DevOps
• Proficiency with version control workflows (Git), GitOps methodologies, automated testing frameworks for infrastructure code, and policy-as-code practices with consistent use of AI assist tools (e.g., Claude, GitHub Copilot) to accelerate automation and troubleshooting

Cloud & Container Orchestration

• Demonstrated experience designing and managing infrastructure across cloud platforms (AWS, Azure, or GCP) with multi-cloud experience highly valued
• Strong expertise with containerization technologies (Docker) and container orchestration platforms (Kubernetes, EKS, AKS, or GKE) including advanced concepts like operators, custom resources, and cluster management
• Ability to design and implement event-driven architectures using cloud-native services (AWS EventBridge, Azure Event Grid, Pub/Sub) and messaging systems with understanding of service mesh technologies (Istio, Linkerd) and API gateway patterns
• Knowledge of networking concepts in cloud and containerized environments including CNI plugins, ingress controllers, load balancing, and service discovery with familiarity in edge computing deployments and hybrid cloud architectures

Observability, Reliability & Security

• Ability to implement comprehensive observability solutions including metrics collection (Prometheus, CloudWatch), distributed tracing (Jaeger, Tempo), and centralized logging (ELK Stack, Loki, Splunk)
• Understanding of Site Reliability Engineering (SRE) principles including SLOs, SLIs, error budgets, and incident response with ability to design and implement chaos engineering practices to improve system resilience
• Experience implementing security best practices including secrets management (Vault, AWS Secrets Manager), vulnerability scanning, and DevSecOps tooling
• Knowledge of disaster recovery strategies, backup automation, and business continuity planning with understanding of compliance frameworks and ability to implement automated compliance controls

Data Platform Operations & ML Infrastructure

• Understanding of cloud-native data pipeline architectures and ETL/ELT orchestration (AWS Glue, Azure Data Factory, Airflow, Prefect) with ability to build and maintain infrastructure supporting ML pipelines, model training workflows, and MLOps practices
• Knowledge of deploying and operating cloud-based data storage systems and platforms (S3, Redshift, Delta Lake, PostgreSQL, MongoDB, OpenSearch, Snowflake)
• Understanding of distributed data processing frameworks (Spark/Databricks, Kafka, Flink) with experience operating Kubernetes-based platforms for data workloads including Spark on K8s, Ray clusters, or Kubeflow
• Ability to implement infrastructure supporting large-scale data systems with appropriate monitoring, cost optimization, and performance tuning including storage tiering, data lifecycle management, and compute resource optimization

Collaboration & Operations

• Strong problem-solving abilities with experience troubleshooting complex distributed systems spanning applications, infrastructure, and data layers
• Excellent communication skills to collaborate effectively with software engineers, data scientists, security teams, and business stakeholders with ability to create clear, comprehensive documentation for infrastructure designs, runbooks, and disaster recovery procedures
• Demonstrated capacity to manage multiple infrastructure initiatives simultaneously while maintaining high availability and reliability standards with proven ability to mentor team members on DevOps practices and operational excellence
• Experience participating in on-call rotations, incident response, and post-mortem processes with ability to balance tactical operational needs with strategic infrastructure improvements

National Interest Project Examples

• Detect and prevent smuggling of drugs and contraband at ports of entry [Link]
• Develop large data pipelines to thwart funding for terrorists, nuclear proliferators, drug cartels, and rogue leaders [Link]
• Applying big data solutions to national security problems [Link]
• Applying image classification for nuclear forensics analysis [Link]
• Develop capabilities for scalable geospatial analytics [Link]

This position is based in Richland, WA or Seattle, WA and requires an onsite presence Monday through Thursday, with Friday as required by business needs.

Qualifications

Minimum Qualifications:

• PhD and 1 year of software engineering experience -OR-
• MS/MA and 3 years of software engineering experience -OR-
• BS/BA and 5 years of software engineering experience -OR
• AA and 14 years of software engineering experience in designing, architecting, programming, deploying, and automating software solutions in support of scientific research or consumer digital product development -OR-
• HS/GED and 16 years of software engineering experience in designing, architecting, programming, deploying, and automating software solutions in support of scientific research or consumer digital product development

Preferred Qualifications

• Degree in computer science, software engineering, or related field
• Demonstrated ability to contribute to technical direction and independently structure complex problems into actionable work, in collaboration with senior engineers and cross-functional teams
• Proven expertise in Python and proficiency in at least one other language (C#/.NET, C++, Go)
• 3-5 years of hands-on DevOps, Platform Engineering, Site Reliability Engineering, or Infrastructure Engineering experience
• Demonstrated contributions to open-s