Overview
The Senior Data Engineer will play a crucial role in supporting a data transformation programme for the financial services sector. This position focuses on building and migrating enterprise data pipelines, specifically transitioning workloads to Microsoft Fabric, while collaborating with Agile delivery teams to ensure successful implementation.
Responsibilities
- Build and enhance metadata-driven ingestion frameworks.
- Develop incremental loads, watermarks, and merge processes.
- Migrate ETL workloads to Microsoft Fabric Pipelines.
- Integrate cloud platforms with on-premises environments.
- Develop reusable deployment patterns and technical documentation.
- Support CI/CD across DEV, TEST, UAT, and PROD environments.
- Test pipelines, resolve defects, and support UAT.
- Work collaboratively within Agile delivery teams.
Requirements
- Strong experience with Microsoft Fabric and Fabric Pipelines.
- Proficient in Azure Data Factory and ADLS.
- Experience with enterprise ETL and data ingestion processes.
- Familiarity with metadata-driven frameworks and incremental loading.
- Proficient in SQL and Python programming languages.
- Experience with CI/CD and multi-environment deployments.
- Skilled in data mapping and pipeline testing.
- Previous experience in banking or financial services is preferred.