Overview
As a Databricks Engineer, you will be responsible for building, maintaining, and optimizing data pipelines using PySpark, SQL, and Delta Lake to prepare raw data for business use. You will work closely with business stakeholders and the Platform team to transform requirements into technical solutions while ensuring data integrity and compliance through monitoring and governance controls. This remote role offers the opportunity for occasional site visits to London or Birmingham and emphasizes collaboration and documentation throughout the project.
Responsibilities
- Build and optimize Databricks data pipelines using PySpark, SQL, and Delta Lake.
- Manage production deployments through Azure DevOps, including code reviews and testing.
- Monitor and troubleshoot data pipelines to resolve issues and optimize performance.
- Implement data quality checks and governance controls using Unity Catalog.
- Collaborate with stakeholders to translate requirements into technical solutions.
- Maintain clear documentation and support runbooks for ongoing operations.
Requirements
- Proven commercial experience as a Databricks Engineer, Data Engineer, or Data Platform Engineer.
- Strong technical expertise in Databricks, PySpark, SQL, and Delta Lake.
- Experience with Azure DevOps, including CI/CD pipelines and automated testing.
- Knowledge of Unity Catalog and data governance practices.
- Ability to solve problems and communicate effectively with technical teams.