Data Engineer

Overview

The Data Engineer will play a crucial role in a Microsoft Fabric implementation for a housing association, focusing on reviewing and executing existing architectural designs from a third party. This position involves collaboration with various stakeholders to enhance data processes and establish best practices within the new data environment, leveraging prior experience in similar implementations.

Responsibilities

  • Conduct a technical review of the existing Microsoft Fabric architecture and data pipelines.
  • Assess and recommend improvements to the current Lakehouse architecture.
  • Design and implement scalable Fabric Medallion Architecture patterns.
  • Build and optimise data pipelines using various Fabric tools and endpoints.
  • Develop robust data engineering frameworks to enhance data reliability and maintainability.
  • Establish best practices for security, governance, and data quality within Microsoft Fabric.
  • Collaborate with BI, Analytics, IT, and business stakeholders to align on data requirements.
  • Support the migration of data into the enterprise data platform.

Requirements

  • Proven experience with Microsoft Fabric implementation and architecture review.
  • Strong background in designing and optimising data pipelines.
  • Familiarity with data ingestion, transformation, modelling, and reporting processes.
  • Experience with Fabric Data Factory, Dataflows Gen2, Spark Notebooks, and SQL endpoints.
  • Understanding of security, governance, lineage, and data quality best practices.
  • Ability to provide technical leadership and mentoring to teams.
  • Experience in greenfield implementations is advantageous.