Overview
The Data Engineer will be responsible for developing and implementing a robust data lake architecture on Azure, utilizing Databricks to facilitate data ingestion, processing, and reporting for the insurance sector. This fully remote, 2-3 month contract role involves collaborating closely with data teams to establish governance, metadata, and access controls, as well as building data transformation logic and API connectors using Python and PySpark.
Responsibilities
- Define and build a data lake architecture on Azure (ADLS + Databricks).
- Implement a layered data model (bronze/silver/gold) for data ingestion and processing.
- Develop ingestion patterns from API sources, supporting both batch and streaming.
- Establish data governance, metadata standards, and access controls.
- Define storage structures, naming conventions, and schema designs.
- Develop API connectors and data ingestion pipelines using Python.
- Implement data transformation logic using PySpark/Python, focusing on data cleansing and validation.
- Configure monitoring, logging, and alerting for data pipelines.
Requirements
- Proven experience with Azure Databricks and data lake architecture.
- Strong knowledge of Python programming and API integrations.
- Experience developing data ingestion patterns for batch and streaming data.
- Familiarity with data governance and metadata management practices.
- Experience with schema design and data transformation using PySpark.
- Prior experience in the financial services sector, ideally within insurance.
- Ability to configure monitoring and testing for data pipelines efficiently.