Overview
This role of Azure Data Engineer focuses on supporting a prominent public sector organisation within a high-profile data and analytics programme. The successful candidate will work remotely, collaborating closely with a multidisciplinary data team to design, develop, and optimize scalable data solutions on the Azure platform, utilizing technologies such as Azure Databricks and Docker. The role is critical in enhancing the organisation's data capabilities for environmental, scientific, and operational decision-making.
Responsibilities
- Design, build and maintain scalable data pipelines and engineering solutions.
- Develop and optimise Azure-based data platforms and services.
- Deploy data engineering solutions within a containerised Docker environment.
- Support the ingestion, transformation, and management of structured and unstructured datasets.
- Build robust ETL and ELT processes for analytics and reporting.
- Collaborate with data scientists, analysts, and business stakeholders to meet data requirements.
- Manage and optimise cloud-based data storage solutions.
- Ensure adherence to data quality, integrity, governance, and security standards.
Requirements
- Proven experience as a Data Engineer in complex data environments.
- Strong expertise in designing and developing Azure-based data solutions.
- Hands-on experience with Azure Databricks, Azure Data Lake, and related Azure technologies.
- Experience developing within containerized environments using Docker.
- Strong SQL and data transformation skills.
- Experience with ETL/ELT pipelines and handling large datasets.
- Understanding of data governance and metadata management principles.
- Familiarity with Power BI, ArcGIS, or geospatial data platforms is highly desirable.