Overview
As a Senior Data Engineer specializing in Databricks, you will be instrumental in building a new SaaS data platform from the ground up in an office-first environment transitioning to a hybrid model. You will collaborate closely with a growing technology team to establish core engineering standards and deliver efficient data pipelines, ensuring production readiness and supporting the ongoing success of the platform.
Responsibilities
- Stand up Azure Databricks as code including workspaces, Unity Catalog, and permissions.
- Build reusable, config-driven frameworks for data ingestion, medallion layers, data quality, and publishing.
- Deliver end-to-end data pipelines from source systems to a modeled gold layer published to Azure SQL.
- Implement CDC ingestion with full history tracking.
- Set engineering standards encompassing CI/CD, testing, environment promotion, scheduling, and alerting.
- Make the platform production-ready through runbooks, backfills, recovery, governance, lineage, and cost control.
- Support go-lives and stabilize pipelines post-launch.
- Coach other engineers to confidently manage pipelines independently.
Requirements
- 6+ years in data engineering with 3+ years of experience specifically with production pipelines on Databricks.
- Deep, current knowledge of Databricks features including Unity Catalog and Delta Lake.
- Strong experience with ingestion methods such as Lakeflow Connect and CDC from SQL Server.
- Proven capability in building frameworks that enhance engineering workflows.
- Experience using AI-assisted development practices effectively in production environments.
- Availability and preference for in-person collaboration in the Bristol office.
- Eligibility for UK SC clearance is required.