Sr Data Engineer, Data Governance

Build and Optimize Data Infrastructure: Develop, construct, test, and maintain large-scale data ingest architecture consisting of diverse cloud-based services (messaging, storage, Kubernetes, persistent data store, serverless functions, etc). Create tooling like SDK, APIs to enable user self-service. Contribute to the design and evolution of our core data platform,ensuring its scalability, reliability, and cost-effectiveness. Implement robust monitoring, alerting, and logging solutions for data pipelines and infrastructure to proactively identify and resolve issues. Design and Implement Scalable Data Pipelines: Design and implement highly reliable and efficient ETL/ELT processes to ingest, transform, and load data from diverse sources (e.g., real-time events, third-party APIs, rich media datasets) into our data lake and data warehouses. Ensure Data Quality and Governance: Implement data validation, cleansing, and reconciliation processes to ensure the accuracy and integrity of our data assets. Work closely with stakeholders (research, engineering, product, and peers more broadly) to translate their data needs into robust data solutions. Provide technical leadership and mentorship to junior data engineers, fostering a culture of technical excellence and continuous learning. Contribute to the evolution of our data architecture and engineering best practices. Extensive Experience: 5+ years of experience in data engineering, with a strong focus on building and maintaining large-scale data pipelines and infrastructure. Programming Proficiency: Expert-level proficiency in at least one major programming language such as Python, Scala, or Java.- Distributed Data Processing: Deep experience with distributed data processing frameworks (e.g., Apache Spark, Apache Beam). Strong foundation in event-based approaches and systems including messaging/topics, pub/sub, queues, etc. Data Warehousing/Lakes: Hands-on experience with data warehousing solutions (e.g., Databricks, Snowflake, Redshift, BigQuery) and data lake technologies (e.g., S3, HDFS). Deep experience with managing large scale, heterogeneous datasets on Databricks is highly preferred. SQL Mastery: Advanced SQL skills for data manipulation, analysis, and optimization. Cloud Platforms: Strong experience with one or more major cloud providers (AWS, GCP, Azure) and their data-related services. Orchestration and DevOps: Familiarity with containerization and orchestration technologies (e.g., Docker, Kubernetes). Proficient at CI/CD-based deployment. Database Knowledge: Solid understanding of relational and NoSQL databases. Data Modeling: Expertise in data modeling, schema design, and data architecture principles. Problem-Solving: Excellent analytical and problem-solving skills, with a track record of tackling complex data challenges. Communication: Strong communication and interpersonal skills, with the ability to collaborate effectively with cross-functional teams. Master's degree in Computer Science, Data Science, or a related quantitative field.