Overview
We are seeking an experienced Data Engineer to manage the construction of a data pipeline for consolidating bordereaux reporting data into a standardized, reporting-ready format. This hands-on role is part of the BI team and involves working independently to build Azure Databricks-based data solutions that support comprehensive reporting and analytics efforts. The contractor will collaborate closely with various stakeholders to ensure the pipeline is robust, efficient, and production-ready.
Responsibilities
- Take ownership of the end-to-end build of a scalable ETL pipeline in Azure Databricks.
- Design and implement data ingestion processes for multiple file formats from shared mailboxes.
- Develop Databricks notebooks, jobs, and pipelines to facilitate data transformation and orchestration.
- Build and implement data validation, quality controls, logging, monitoring, and error handling systems.
- Integrate solutions with Azure DevOps CI/CD pipelines across development, test, and production environments.
Requirements
- Strong hands-on experience as a Data Engineer with 5+ years in the field.
- Proven expertise in Azure Databricks, particularly with ETL pipelines and Delta Lake architecture.
- Strong proficiency in Python, PySpark, and Spark SQL.
- Experience with Unity Catalog and Azure Databricks governance features.
- Familiarity with implementing CI/CD pipelines using Azure DevOps.
- Experience supporting BI/reporting layers such as Power BI is advantageous.
- Knowledge of the insurance sector is a plus.