Overview
We are seeking a highly skilled Senior Data Engineer with extensive expertise in Databricks to join a dynamic technology team focused on developing an innovative, Azure-based SaaS platform. In this collaborative role, you will be instrumental in building and deploying the Databricks data platform from scratch, while establishing scalable engineering practices and delivering production-ready data pipelines. This position is well-suited for individuals who enjoy tackling complex data challenges and working closely with engineering teams to create robust platforms.
Responsibilities
- Design, build and deploy an Azure Databricks platform, including Unity Catalog, permissions, and environments.
- Develop scalable, reusable data pipelines covering ingestion, transformation, data quality, and publishing.
- Build and optimise Delta Lake pipelines using PySpark and Spark SQL.
- Implement CI/CD, automated testing, monitoring, and data governance.
- Develop reusable frameworks to support future deployments.
- Ensure platform security, reliability, performance, and cost efficiency.
- Collaborate with technical leads, architects, and engineers to deliver production-ready solutions.
- Use AI-assisted development tools to accelerate delivery while maintaining engineering quality.
Requirements
- 6+ years' data engineering experience, including 3+ years working with Databricks in production environments.
- Strong hands-on expertise in Azure Databricks, Unity Catalog, Delta Lake, PySpark, and Spark SQL.
- Proven experience building Databricks platforms from initial setup through to production.
- Experience with Databricks Asset Bundles, CI/CD, incremental loading, and Delta MERGE.
- Strong knowledge of data ingestion, change data capture (CDC), and medallion architecture.
- Experience building automated, reusable data engineering frameworks.
- Strong understanding of data quality, governance, security, and monitoring.
- Practical experience using AI-assisted development tools.