Job Title: AI & Big Data Engineering Specialist

Location: India

Timings : 6:30 PM to 3:30 PM IST

Department: Analytics Engineering

Duration: Until EOY (project is until 2030).

Role Overview

We are seeking a highly skilled AI & Big Data Engineering Specialist to design and implement scalable data pipelines and advanced analytics solutions leveraging machine learning and agent-based automation. This role will focus on building intelligent systems that integrate with modern big data platforms such as Databricks, Snowflake, and AWS, enabling predictive insights and automation within our enterprise data platform.

Key Responsibilities

• Design and develop large-scale data pipelines using Spark and Databricks or Snowflake for ingestion, transformation, and integration of structured and unstructured data.
• Build and deploy machine learning models for anomaly detection, predictive analytics, and advanced insights using Databricks MLlib, scikit-learn, or Snowpark ML.
• Develop AI agents leveraging Databricks Agent Framework for workflow automation and intelligent decision-making.
• Implement feature engineering and model lifecycle management using MLflow and Databricks Feature Store.
• Optimize data workflows for performance, scalability, and cost efficiency across AWS, Databricks, and Snowflake environments.
• Collaborate with analytics and business teams to translate requirements into technical solutions.
• Stay current with emerging AI/ML technologies and integrate them into our data platform solutions.

Required Skills & Qualifications

• Strong programming skills in Python and SQL.
• Hands-on experience with Databricks ML features (Delta Lake, MLflow, AutoML, Feature Store).
• Proficiency in Spark for big data processing.
• Experience with machine learning techniques (classification, regression, anomaly detection).
• Familiarity with agent-based automation frameworks (Databricks Agent Bricks or similar).
• Expertise in AWS services (S3, Glue, Lambda, IAM) and Snowflake for data integration.
• Solid understanding of data architecture and distributed systems.

Preferred Qualifications

• Experience in pharma or healthcare data analytics and engineering.
• Knowledge of Generative AI and LLM integration for data insights.
• Familiarity with Snowpark ML for ML in Snowflake.
• Exposure to real-time data processing and streaming analytics.

Soft Skills

• Excellent problem-solving and analytical thinking.
• Ability to translate business requirements into technical solutions.
• Strong communication and collaboration skills.