Overview
We are seeking a Data Engineer to contribute to a UK public sector project, focusing on the delivery of key data components within an evolving data ecosystem. This role involves close collaboration with architects and delivery teams to build and maintain data pipelines and models using Azure and Databricks. The position offers technical autonomy, ownership, and the opportunity to enhance data flow efficiency and performance standards.
Responsibilities
- Build and maintain ETL pipelines and code in cloud data platforms.
- Support ingestion activity and onboarding of new data sources.
- Develop Data Engineering workloads using the Databricks Data Intelligence Platform.
- Construct raw, refined, and curated data layers and catalogue assets.
- Validate solutions against functional and non-functional requirements.
- Deliver datasets, transformations, and performance-optimised data products.
- Monitor and measure pipeline performance; support incident resolution.
- Engage in Agile ceremonies and governance forums.
Requirements
- Strong experience with Python, PySpark, and SQL for data engineering.
- Hands-on experience with Azure Databricks.
- Knowledge of Azure Data Factory and Azure Fabric.
- Experience with Microsoft Purview and event-driven data ingestion.
- Understanding of SOLID principles, async programming, and design patterns.
- Experience in unit and integration testing in Databricks.
- Familiarity with secure ETL design and Azure best practices.
- Prior experience working within UK public sector environments.