Overview
The Lead Data Engineer will spearhead a team of 5 to 7 engineers, guiding the transition of data products into a new governed Databricks workspace. This role balances leadership responsibilities with hands-on engineering tasks, ensuring collaboration between data teams and business stakeholders to deliver quality data products that cater to various operational needs.
Responsibilities
- Lead and coach a team of data engineers, setting technical direction and driving delivery.
- Collaborate with business stakeholders and data product owners to prioritize and define project requirements.
- Engage in hands-on data modelling, pipeline design, and code reviews to ensure high standards of quality.
- Develop reusable Databricks templates and libraries to enhance data product efficiency.
- Prepare data products for integration with AI tools such as Databricks Genie.
- Establish standards for testing, CI/CD, and data quality across the team.
Requirements
- Minimum of 7 years of experience in data engineering, including at least 2 years in a leadership or mentoring role.
- Strong hands-on expertise with Databricks, Spark or PySpark, Python, and SQL.
- Demonstrated ability to translate business priorities into technical solutions.
- Proficient in data modeling and building production data products.
- Experience working with Unity Catalog, Delta Lake, and Delta Live Tables is essential.
- Fluent in English; proficiency in French is a plus but not required.