Data Engineer-ETL,Data factory

Job description

• Develop and manageETL workflowsusingAzure Data Factory (ADF).
• Design and implementdata pipelinesusingPySpark on Azure Databricks.
• Work withAzure Synapse Analytics, Azure Data Lake, and Azure Blob Storagefor data ingestion and transformation.
• OptimizeSpark jobsfor performance and scalability inDatabricks.
• Automatedata workflowsand implementerror handling & monitoringin ADF.
• Collaborate withdata engineers, analysts, and business teamsto understand data requirements.
• Implementdata governance, security, and compliancebest practices in Azure.
• Debug and troubleshootPySpark scripts and ADF pipeline failures.
• 4+ yearsof experience inETL developmentwithAzure Data Factory (ADF).
• Hands-on experience withAzure DatabricksandPySparkforbig data processing.
• Strong knowledge ofAzure services
• Proficiency inPythonandPySparkfor data transformation and processing.
• Experience withCI/CD pipelinesfor deploying ADF pipelines and Databricks notebooks.
• Strong expertise inSQLfor data extraction and transformations.
• Knowledge ofperformance tuning in Sparkandcost optimization on Azure.

Skills

Azure Data Factory,Pyspark,Azure