Overview
The Data Engineer will play a critical role in the greenfield development of a cloud data warehouse and portfolio monitoring platform for a private equity firm. Working as a senior member of the team, the contractor will leverage Azure and Databricks to automate manual workflows and build robust data pipelines that will lay the groundwork for future fund launches.
Responsibilities
- Lead the development of a cloud data warehouse using Azure and Databricks.
- Automate manual Excel workflows into efficient, auditable data pipelines.
- Implement a medallion lakehouse architecture (Bronze/Silver/Gold).
- Collaborate with team members to define data engineering best practices.
- Utilize SQL and Python (PySpark) to manipulate and analyze data.
Requirements
- Proven experience with Azure cloud services.
- In-depth knowledge of Databricks.
- Familiarity with medallion architecture in data lakes.
- Background in finance, preferably within private equity or asset/investment management.
- Strong data engineering skills, particularly in SQL and Python (PySpark).
- Demonstrated ability to work independently and lead projects.