Overview
The Azure Data Engineer will play a crucial role in designing, developing, and optimizing data pipelines within an Azure environment. The contractor will collaborate with cross-functional teams, leveraging their expertise in Azure technologies to support data transformation and analytics in the Consumer Goods and Retail sectors. This position is focused on building reliable data solutions that drive business insights and decision-making.
Responsibilities
- Design and develop scalable ETL/ELT data pipelines to process large volumes of data.
- Implement data transformation solutions using PySpark and Azure Databricks.
- Orchestrate and monitor data pipelines with Azure Data Factory, managing scheduling and error handling.
- Integrate data from various commercial sources, creating analytics-ready datasets.
- Ensure data quality and implement validation and logging controls for reliable data management.
- Collaborate with analytics and BI teams to meet commercial data requirements.
- Optimize data processing and engineering practices for performance.
Requirements
- 6+ years of experience in Data Engineering.
- 3+ years of hands-on experience with Azure and Databricks.
- Proficiency in Azure Data Factory, PySpark, and SQL.
- Experience in the Consumer Goods or Retail sector.
- Strong understanding of commercial data including Sales, Customer, and Pricing data.
- Ability to create curated datasets for reporting and analytics.
- Knowledge of data quality, validation, and monitoring practices.
- Relevant Microsoft Azure certification preferred.