Lead Azure Databricks Engineer

Overview

At PNNL, we focus on advancing scientific research through innovative technology and operations. Our Computing and Information Technology organization is on a mission to redefine the digital workplace, empowering our researchers by streamlining operations and leveraging cutting-edge technologies.

Digital Platforms at PNNL partners with various departments to engineer innovative software solutions and robust analytics capabilities. By employing cloud technologies, DevSecOps, and AI, we aim to modernize existing platforms and create novel solutions that meet the challenges of tomorrow.

Responsibilities

We are looking for a dynamic Lead Azure Databricks Engineer to spearhead the design, construction, and operation of our data lakehouse, which serves as a crucial analytics engine for PNNL Enterprise Systems. Your role will be pivotal in delivering reliable and efficient data products for our ERP and other enterprise applications while enabling self-service analytics through tools like Power BI and AI/ML.

You will act as the technical backbone of our lakehouse within a data transformation initiative, crafting a Databricks-based architecture that confidently moves data from source systems to optimized gold tables. Using the medallion architecture (Bronze/Silver/Gold) with Delta Lake and Unity Catalog, you’ll build resilient pipelines that turn raw data into analytical assets while emphasizing efficient delivery alongside a vision for future development.

Your expertise will contribute to the modernization of legacy data warehouses and ETL tools, migrating them to Azure Databricks and refactoring outdated jobs into scalable frameworks. You will implement CI/CD solutions for Databricks (e.g., DAB, Azure DevOps, GitHub Actions) and standardize deployment practices across different environments. Your duties include configuring and managing workspaces, clusters, jobs, and workflows; optimizing for performance and reliability; and integrating data quality, monitoring, and observability to maintain healthy pipelines.

Security and governance will be fundamental to your responsibilities. You will implement role-based access controls, data masking, and fine-grained models with Unity Catalog, promoting secure data sharing while ensuring proper classification and traceability.

As a collaborator and mentor, you’ll guide engineers and analysts in adopting best practices for the lakehouse and contemporary data engineering standards, including coding, testing, version control, and documentation. Staying abreast of Azure and Databricks advancements, you will recommend and pilot new features such as Delta Live Tables and Unity Catalog enhancements to build a secure and sustainable data platform that accelerates scientific research and mission success.

Qualifications

Minimum Qualifications:

• PhD with 3 years of relevant experience OR
• MS/MA or higher with 5 years of relevant experience OR
• BS/BA with 7 years of relevant experience OR
• AA with 16 years of relevant experience OR
• HS/GED with 18 years of relevant experience
• Experience in software development related to scientific research or consumer digital product development may be recognized.

Preferred Qualifications

• 7+ years in data engineering or platform engineering with 3-5+ years in cloud data platforms.
• 5+ years of experience with Azure Databricks production, including Delta Lake, SQL, notebooks, Jobs/Workflows, and cluster management.
• 3-5+ years focused on designing and operating ingestion-to-gold pipelines for ERP or complex transactional sources.
• Experience with Azure Data Factory and/or Fabric Data Pipelines for integration and orchestration.
• Familiarity with core Azure services: ADLS Gen2, Key Vault, Azure DevOps or GitHub.
• Proficiency in Python and SQL within a Spark/Databricks environment.
• Experience implementing Databricks Asset Bundles (DAB) for standardized CI/CD workflows.
• Familiarity with GenAI / LLM-based tools for task automation.
• Exposure to agentic AI is a plus.

Work Environment

This position involves access to sensitive systems and information. U.S. Citizenship is required. Pacific Northwest National Laboratory is subject to specific access requirements for technologies and information.

About PNNL

Pacific Northwest National Laboratory (PNNL) is a renowned research institution committed to integrity, creativity, collaboration, impact, and courage. Located in eastern Washington State, PNNL offers an inspiring research environment along with attractive benefits, including flexible work schedules, comprehensive health insurance, and opportunities for personal and professional growth.

Commitment to Excellence and Equal Employment Opportunity

PNNL values a diverse workforce and is committed to fair treatment and equal opportunity in employment. We encourage applications from all qualified individuals, promoting diversity in our workplace.