Lead Staff Systems Reliability Engineer (Linux & Distributed Systems)

What we do 

We are looking to hire a Lead Systems Reliability Engineer to join our engineering team to continue building and maintaining our data-driven platform. We leverage technologies like Aerospike, MongoDB, and Kafka to perform many real time activities, translating to with a p99 latency under 1 millisecond on the back end!

Do you enjoy tuning, performance testing, troubleshooting, automation, and operating at scale? Does testing next-gen hardware, evaluating data access patterns, and designing automation around distributed systems excite you?

What makes this role different:

• First in the Industry: The Trade Desk is the first company to run over 5MM QPS to NVMe in Aerospike on a single node, forcing core software redesigns to achieve this scale.
• Work on Cutting-Edge Hardware: Design clusters with nodes featuring 300TB of NVMe, 3TB RAM, and 512 cores, delivering a global 2,500GB/s throughput directly from flash.
• Shape the Future of Infrastructure: Spec your own systems and collaborate directly with AMD and NoSQL vendors to run PoCs and optimize bleeding-edge technology for internet-scale workloads.
• Deep Performance Engineering: Dive into kernel, hardware, and system interactions, leveraging tools like flamegraphs, NUMA counters, BIOS tuning, and synthetic testing to achieve world-class performance.
• Push Hardware Endurance Limits: Build clusters engineered to withstand over 1 zettabyte of endurance.

What you’ll do: 

• Lead a team to influence, manage, and plan work streams, systems, and data structures at scale within a global ecosystem, spanning multiple infrastructure providers (cloud and traditional datacenters).
• Encourage, improve, and build infrastructure automation in a way that works with stateful systems at scale.
• Own operations for Linux-based systems running Aerospike, Kafka, and Mongo.
• Serve as a point of contact to review new use cases, answer questions, and participate in on-call rotation.
• Learn to be a NoSQL SME. You do not need experience to apply – we will train you.
• Benchmark and analyze next generation hardware offerings.

Who you are: 

Skills and Experience

• Linux operating system
• Leadership experience and ability to mentor
• Troubleshooting
  • Techniques for isolation, scientific method
  • Identify bottlenecks (Is it CPU? IO?)
• Nice-To-Have experience:
  • Physical hardware (on-prem) internals, management, and operation
  • Performing testing and tuning
  • Databases (relational or NoSQL)
  • Ansible/PyInfra/Chef
  • Prometheus
  • Kubernetes
  • Python/Ruby/Rust/Bash/Golang/C#

• Thinking beyond the task at hand to deeply understand the 'why' behind an objective.
• A welcoming of ideas, and understanding of, perspectives that are different from your own and an interest in seeking and building from a common ground.
• You are a creative thinker, not bound by "the way things have always been done" but are thinking of the questions nobody has thought of and are "yet to be asked". What you know is less important than how well you learn, innovate, collaborate, and adapt.
• As a global team from many diverse backgrounds, experiences, and perspectives, you value and seek out paths for fostering diversity.

The Trade Desk does not accept unsolicited resumes from search firm recruiters. Fees will not be paid in the event a candidate submitted by a recruiter without an agreement in place is hired; such resumes will be deemed the sole property of The Trade Desk. The Trade Desk is an equal opportunity employer. All aspects of employment will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, veteran status, or any other status protected under federal, state, or local law.