Overview
We are seeking a Senior Data Engineer with extensive experience in Azure Databricks to contribute to the design, development, and optimization of modern data platforms. The contractor will build scalable data pipelines and manage large datasets, collaborating with data teams to deliver high-quality solutions within an Azure environment.
Responsibilities
- Design and develop scalable data pipelines aligned with business requirements.
- Optimize ETL/ELT processes using Python, PySpark, and SQL/Spark SQL.
- Implement and maintain Lakehouse architectures on Azure Data Lake Storage Gen2.
- Utilize Azure Databricks features, including job clusters and notebooks, for efficient data processing.
- Orchestrate workflows using Lakeflow Jobs and Pipelines.
- Manage data governance and access control using Unity Catalog.
- Facilitate CI/CD processes with Git-based version control, ideally within Azure DevOps.
- Support the ingestion of data using modern approaches such as Change Data Capture (CDC).
Requirements
- Proven expertise in Python, PySpark, and SQL/Spark SQL for data pipeline construction.
- Solid knowledge of Apache Spark, including performance optimization techniques.
- Hands-on experience with Azure Databricks and its various components.
- Familiarity with Lakehouse architecture and data layers (Bronze, Silver, Gold).
- Experience with data ingestion methods, including Auto Loader and incrementally loaded data.
- Proficient in orchestrating workflows using Lakeflow frameworks.
- Understanding of Unity Catalog for data governance in a Lakehouse environment.
- Experience with CI/CD tools, particularly in an Azure DevOps environment.