Overview
We are seeking a Data Engineer to spearhead the development of a Databricks environment for a private equity firm poised to launch its largest fund yet. This pivotal role involves building and optimizing the data platform while collaborating closely with key stakeholders across various teams including Finance and Compliance to enhance data visibility and integrity. The individual will have the opportunity to directly impact the firm's data operations and strategic initiatives.
Responsibilities
- Design and implement the firm's first Databricks instance to establish a medallion architecture on Azure.
- Integrate the new data platform with existing portfolio administration and data intelligence systems.
- Develop a golden data source that meets the firm's analytical needs.
- Optimize data performance through techniques such as ACID transactions and performance tuning.
- Engage with senior stakeholders to align technical solutions with business requirements.
- Diagnose and resolve data issues through effective troubleshooting and data management strategies.
Requirements
- Proven experience delivering Databricks and Delta Lake builds, including performance tuning techniques.
- Strong understanding of medallion architecture and proficiency in PySpark, including key functions like lazy evaluation and partitioning.
- Solid background in Azure data platform technologies and familiarity with Azure Analysis Services is advantageous.
- Exceptional communication skills to liaise effectively with senior, non-technical stakeholders.
- Experience in Private Equity or a similar financial services environment is preferred.