Overview
We are seeking a skilled Azure Data Engineer to join our team on a contract basis. The successful candidate will collaborate with data teams to design and implement efficient data solutions, utilizing Azure technologies and frameworks. The primary focus of this role will be on developing and optimizing data pipelines to support data processing and governance efforts.
Responsibilities
- Develop and optimize data pipelines using Azure Databricks, PySpark, and ADF.
- Build scalable data solutions utilizing Azure Data Lake Storage (ADLS), Python, and SQL.
- Implement data transformation, cleansing, and processing techniques with PySpark.
- Manage data governance, security, and access control using Unity Catalog.
- Troubleshoot and optimize data pipelines for improved performance and reliability.
Requirements
- Proven experience with Azure Databricks, ADF, and PySpark.
- Strong proficiency in SQL and Python programming.
- Expertise in building solutions using Azure Data Lake Storage (ADLS).
- Familiarity with Unity Catalog for data governance and security.
- Capable of developing and optimizing complex data pipelines.