Overview
We are looking for a Lead Test Engineer to take charge of the testing strategy and quality engineering initiatives within a multi-disciplinary team focused on the Azure Databricks platform. This role requires an experienced professional who will shape testing practices, implement a structured test framework, and champion data quality tooling while collaborating closely with test and data engineers.
Responsibilities
- Define and own the test strategy for the Azure Databricks data platform.
- Design and implement a structured test framework to establish consistent testing patterns and reusable components.
- Build CI/CD pipelines in Azure DevOps that automate test execution and identify issues before production.
- Introduce test-driven development (TDD) practices within the engineering team.
- Define data quality checks and validation rules using tools like Soda Data Quality or DQX.
- Lead and mentor test engineers while enhancing testing standards across the platform.
- Collaborate with data engineers and architects to integrate testing early in the development lifecycle.
- Document the test strategy, framework, and standards for ongoing reference by the engineering team.
Requirements
- Proven experience defining and delivering a test strategy for a data platform.
- Hands-on experience building test frameworks for Databricks, including pytest-based tests.
- Practical knowledge of test-driven development (TDD) and its implementation in an engineering team.
- Experience building or configuring CI/CD pipelines (Azure DevOps) for automated tests.
- Experience leading or mentoring test engineers to improve quality practices.
- Strong understanding of data pipeline testing challenges such as data quality and schema validation.
- Proven track record of producing clear documentation for testing practices and frameworks.
- Familiarity with UK Public Sector environments or Government Digital Standards is desirable.