Staff Software Engineer, Observability

SUMMARY

We have an opening for a Staff Software Engineer on our Infrastructure Team, with specific focus on Observability - both internal and customer-facing. This is an opportunity to join a mission-critical engineering team that is driving the productivity and reliability of Temporal’s developers and core platforms, respectively. We are a passionate team of talented developers who truly care about our mission and enjoy working deeply across the entire Temporal ecosystem to be a force multiplier across the organization.

What You'll Do

Build:

• Lead the end-to-end Software Development Lifecycle: goals & requirements solicitation, design & review, implementation, operationalization & deployment, support & maintenance.
• Lead feature design, review with stakeholders, iterate to incorporate feedback and drive consensus.
• Clearly document design choices and operational knowledge to successfully deploy and manage the software you develop.
• Provide appropriate test and production readiness coverage for unit, integration, and performance of your feature ownership area.

Own:

• Set a high bar for technical excellence and take pride in the software you develop.
• Design and build multi-component, distributed systems that operate at scale.
• Investigate issues with a methodical approach to identify a root cause.
• Understand performance and reliability implications of design options at scale. Make related tradeoffs.
• Participate in the team’s on-call rotation.

Learn:

• Expert-level knowledge of architecture and services of assigned domain. Strong command over all aspects of the Temporal ecosystem. This role provides a strong opportunity for diverse technical exposure!
• Investigate and understand ways to best leverage Temporal’s own software to power our mission.
• Deeply understand the needs of Temporal internal developers and external customers, and leverage that knowledge for product development and feature design.

Collaborate:

• Lead and mentor fellow team members.
• Participate in design reviews and contribute to design of other features.
• Share design principles for building reliable systems at scale.

What You'll Bring

• User-first mindset. You’re excited by the opportunity to empower others through tooling, and enjoy deeply internalizing user goals and use cases to build effective solutions.
• Motivated by impact. You are driven by a desire to make positive things happen.
• Strong opinions about tools and technology that are equally balanced by a pragmatic drive for impact.
• Ability to work in a self-directed manner in a fast-paced environment.
• Excellent collaboration and communication skills.

Skills & Technologies

• Demonstrated ability to develop horizontally scalable, resilient, and high performance distributed systems in a production environment.
• Experience designing, implementing, deploying, and supporting large scale, geographically distributed observability and/or high throughput data streaming/processing pipelines, or similar.
• Expert in one or more high-level programming languages, preferably Go.
• Expert-level Kubernetes skills.
• Expert-level query development skills, preferably SQL.
• Hands-on experience with one or more cloud providers, preferably AWS, or GCP.
• Thorough understanding of computer architecture, operating systems, and networking.
• Familiarity with best practices regarding monitoring, instrumenting, and configuring infrastructure.

Team Highlights

• Core observability tech stack: Clickhouse, Prometheus, Grafana, Loki, Thanos, Kubernetes, Temporal.
• Our recent observability project had the biggest impact to Temporal margins of any engineering effort in company history while also unlocking unprecedented operator capabilities.
• “Observability is the beating heart of our ability to deliver a mission critical service” -Preeti Somal, Senior VP of Engineering.

COMPENSATION

• The estimated pay range for this role is $212,000 - $286,200 depending on qualifications and location.
• This role is eligible to participate in Temporal's equity plan.