Overview
We are seeking a skilled Data Engineer to lead the development of a new Databricks environment for a private equity firm preparing to launch its largest flagship fund. The contractor will work collaboratively with key stakeholders across various departments to enhance data quality and visibility, creating the firm's first golden data source through the implementation of a medallion architecture on Azure.
Responsibilities
- Design and build the firm's Databricks instance, implementing a medallion architecture on Azure.
- Integrate the new environment with existing portfolio administration and data intelligence platforms.
- Deliver strong hands-on capabilities in Databricks and Delta Lake builds, ensuring efficient ACID transactions and performance tuning.
- Translate technical needs into business requirements for senior, non-technical stakeholders.
- Diagnose partition skew issues and optimize performance through techniques like Z-ordering and VACUUM.
Requirements
- Proven experience delivering Databricks and Delta Lake builds.
- Strong command of medallion architecture concepts and PySpark.
- Solid experience with Azure data platform and related tools.
- Ability to engage with senior, non-technical stakeholders effectively.
- Previous experience in Private Equity or wider Financial Services is highly desirable.