Overview
The Data Engineer will play a crucial role in a Microsoft Fabric implementation for a housing association, focusing on reviewing and executing existing architectural designs from a third party. This position involves collaboration with various stakeholders to enhance data processes and establish best practices within the new data environment, leveraging prior experience in similar implementations.
Responsibilities
- Conduct a technical review of the existing Microsoft Fabric architecture and data pipelines.
- Assess and recommend improvements to the current Lakehouse architecture.
- Design and implement scalable Fabric Medallion Architecture patterns.
- Build and optimise data pipelines using various Fabric tools and endpoints.
- Develop robust data engineering frameworks to enhance data reliability and maintainability.
- Establish best practices for security, governance, and data quality within Microsoft Fabric.
- Collaborate with BI, Analytics, IT, and business stakeholders to align on data requirements.
- Support the migration of data into the enterprise data platform.
Requirements
- Proven experience with Microsoft Fabric implementation and architecture review.
- Strong background in designing and optimising data pipelines.
- Familiarity with data ingestion, transformation, modelling, and reporting processes.
- Experience with Fabric Data Factory, Dataflows Gen2, Spark Notebooks, and SQL endpoints.
- Understanding of security, governance, lineage, and data quality best practices.
- Ability to provide technical leadership and mentoring to teams.
- Experience in greenfield implementations is advantageous.