Overview
We are seeking a skilled Databricks Resident Solutions Architect with active SC clearance to embed with a UK public sector client. The role will focus on designing, governing, and delivering a production-level Databricks Lakehouse, ensuring compliance with UK data sovereignty and security requirements. The contractor will collaborate with both the client and Databricks while leading critical modernization efforts of sensitive datasets.
Responsibilities
- Lead the architecture and build of a secure Databricks Lakehouse on Azure or AWS using medallion design.
- Design and implement Unity Catalog governance including role-based access control and security measures.
- Build production-grade ingestion pipelines with Lakeflow Declarative Pipelines and Workflows.
- Establish CI/CD and platform engineering standards using Databricks Asset Bundles and Terraform.
- Oversee migrations from legacy platforms with clear transition plans.
- Monitor DBU consumption and optimize cluster policies for cost visibility.
- Enable MLOps and GenAI use cases within scope, utilizing MLflow and Model Serving.
- Act as a trusted advisor to stakeholders and provide coaching to client engineers.
Requirements
- Active UK SC clearance, including sponsoring organisation and expiry date.
- Current DPP vetting as a Databricks Resident Solutions Architect or equivalent.
- 4+ years of hands-on experience with Databricks delivery, including two end-to-end production implementations.
- Deep experience with Unity Catalog configuration and role-based access control on sensitive data.
- Strong proficiency in PySpark, Spark SQL, and Delta Lake with performance tuning expertise.
- Production pipeline experience with Lakeflow, DLT, and Workflows.
- Desirable certifications include Databricks Certified Data Engineer Professional or Solutions Architect Champion.
- Understanding of GDS Service Standard and cloud security principles relevant to government data classifications.