Overview
We are seeking a Senior Data Engineer for a 12-month contract role focused on building a greenfield data warehouse and portfolio monitoring platform for a leading private equity firm. The contractor will design, architect, and implement a modern, cloud-based data platform using Databricks on Azure, collaborating closely with business stakeholders and technical teams to ensure the development of robust, scalable data solutions.
Responsibilities
- Design and build a new Databricks environment on Azure from the ground up.
- Define the architecture and technical standards for the data warehouse and portfolio monitoring platform.
- Design and implement data ingestion pipelines from Aztec (SFTP/file-based) and Canoe (API-based).
- Build and maintain scalable ETL/ELT processes using Python and SQL.
- Develop an automated reconciliation framework comparing data sources.
- Implement data quality controls, validation checks, and exception management processes.
- Establish formal QA, testing, and approval gates across the data lifecycle.
- Produce technical documentation and support knowledge transfer activities.
Requirements
- 5+ years' experience in Data Engineering or Data Platform Engineering roles.
- Strong commercial experience with Databricks and Azure.
- Advanced skills in Python development.
- Strong SQL and data modelling experience.
- Proven experience building cloud-based data warehouses and platforms.
- Experience designing and implementing ETL/ELT pipelines.
- Experience integrating data from APIs and file-based/SFTP sources.
- Strong understanding of data quality, governance, and testing frameworks.