Overview
We are looking for a Senior Data Engineer with expertise in Azure, Databricks, PySpark, and SQL to support a UK public sector project. In this role, you will collaborate with architects, analysts, and engineers to develop and optimize data pipelines and models as part of a growing data ecosystem. This position offers the opportunity to work on scalable data solutions, ensuring they meet performance and security standards while actively participating in Agile processes.
Responsibilities
- Build and maintain ETL pipelines and code in cloud data platforms.
- Support ingestion activity and onboarding of new data sources.
- Work with the Databricks Data Intelligence Platform and develop Data Engineering workloads.
- Construct raw, refined, and curated data layers; catalogue assets appropriately.
- Validate solutions against functional and non-functional requirements.
- Deliver datasets, transformations, and performance-optimised data products.
- Improve processes, engineering patterns, and reusable tooling.
- Monitor and measure pipeline performance; support incident resolution.
Requirements
- Strong experience with Python, PySpark, and SQL for data engineering.
- Hands-on experience with Azure Databricks.
- Strong knowledge of Azure Data Factory and Fabric.
- Experience with Microsoft Purview.
- Familiarity with event-driven data ingestion (Event Grid/Pub-Sub).
- Understanding of SOLID principles, async programming, and mediator/factory patterns.
- Experience delivering unit and integration testing in Databricks.
- Knowledge of secure ETL design with Entra IMID/SCIM integration.