Overview
The Data Engineer will focus on extracting and transforming data from Azure DevOps using Databricks to create reliable, reusable datasets for a metrics dashboard serving various technology teams. The role involves collaboration with both engineering and product stakeholders to define metric requirements, ensuring that data quality and validation processes are in place. This hands-on position requires the engineer to build robust data ingestion pipelines and models, contributing to important data-driven decision-making for clients through the consultancy team.
Responsibilities
- Extract data from Azure DevOps using APIs and/or Analytics data.
- Build robust ingestion pipelines into Databricks.
- Transform Azure DevOps work-item and historical data into clean datasets.
- Create reliable data models and tables for dashboard consumption.
- Develop calculations for engineering and delivery metrics.
- Improve data quality, validation, and monitoring processes.
- Collaborate with product and engineering stakeholders to implement metric definitions.
- Create reusable components and approaches for future implementations.
Requirements
- Proven experience with Azure DevOps and its API.
- Strong proficiency with Databricks and data engineering principles.
- Ability to create and manage data ingestion pipelines.
- Experience with building and maintaining data models for analytics.
- Familiarity with data validation and quality improvement techniques.
- Must possess the right to work in the UK.