Overview
The Senior Data Engineer will play a pivotal role in a major Liquidity Reporting programme aimed at delivering regulatory reporting solutions for large UK insurers. This hands-on position involves enhancing a Databricks-based data engineering framework and focusing on the development of scalable PySpark pipelines. Working closely with offshore teams, the contractor will ensure high engineering standards while providing technical leadership and conducting code reviews.
Responsibilities
- Design and develop scalable PySpark pipelines for data ingestion, validation, and transformation.
- Enhance and maintain a Databricks-based data engineering framework.
- Create reusable libraries and ensure high engineering standards are met.
- Provide technical leadership and conduct code reviews for development work.
- Guide and support offshore teams in technical delivery.
- Ensure alignment of delivery quality with architectural and regulatory requirements.
- Collaborate with stakeholders to understand regulatory reporting needs.
Requirements
- Strong experience with Databricks and PySpark.
- Solid knowledge of the Azure data platform.
- Proven ability to build reusable frameworks and libraries.
- Experience in leading code quality initiatives and working with offshore development teams.
- Desirable experience in regulatory reporting or within financial services.
- Exposure to Tagetik or similar reporting tools.
- Knowledge of metadata-driven or configurable data models.