Overview
The Technical Lead Databricks Engineer (Azure) will guide the development of a new Azure data platform tailored for a client with significant growth potential and no existing legacy systems. In this remote role, the contractor will collaborate closely with the client team to translate architectural designs into actionable plans, ensuring high-quality data solutions are implemented effectively and efficiently, while fostering team capabilities through mentoring and knowledge sharing.
Responsibilities
- Lead the engineering build of the Databricks Lakehouse on Azure according to the target architecture.
- Serve as the primary technical contact for the client team, facilitating working sessions and technical discussions.
- Translate architectural decisions into clear delivery plans and communicate effectively with technical and non-technical stakeholders.
- Build and review ingestion and transformation pipelines across medallion layers for batch and streaming data.
- Implement Unity Catalog structure, access controls, and data lineage, and integrate with Microsoft Purview.
- Establish engineering standards for code quality, testing, and deployment processes.
- Automate deployment processes using Databricks Asset Bundles and Azure DevOps or GitHub Actions.
- Mentor client engineers through code reviews and technical training sessions.
Requirements
- Proven experience delivering data solutions directly to clients, preferably in a consultancy or systems integrator setting.
- Strong client-facing skills with the ability to present technical concepts and manage stakeholder relationships.
- Experience as a technical lead or principal engineer on enterprise-scale Databricks platforms on Azure.
- Hands-on data engineering expertise in Python or PySpark, and advanced SQL.
- Familiarity with Delta Lake, Unity Catalog, medallion architecture, and Databricks Workflows.
- Experience with CI/CD for Databricks using Azure DevOps and Terraform or Bicep.
- Understanding of UK GDPR and data governance principles, particularly with Microsoft Purview.
- Experience building greenfield platforms and mentoring engineering teams is desirable.