Overview
We are looking for a Senior Data Engineer to support the implementation of a data strategy for a major financial services client. In this role, you will design and deploy data engineering solutions, including migrating existing ETL processes and onboarding new data sources. You will work in an Agile environment alongside architects, analysts, and business stakeholders to deliver high-quality data solutions.
Responsibilities
- Enhance an existing metadata-driven ADF ingestion framework to onboard new data sources.
- Develop incremental load processes and ingestion mappings.
- Migrate ETL processes from Azure Data Factory and Azure Data Lake Storage to Microsoft Fabric Pipelines.
- Integrate Fabric Pipelines through On-Premises Data Gateways.
- Build parameterised templates and reusable Fabric deployment patterns.
- Produce and maintain source-to-target data mappings and technical specifications.
- Promote build artefacts through DEV, TEST, UAT, and PROD using established CI/CD and DevOps processes.
- Test new and migrated data pipelines, resolving defects as needed.
Requirements
- Strong hands-on experience with Microsoft Fabric, especially Fabric Pipelines.
- Proficient with Azure Data Factory and Azure Data Lake Storage.
- Experience in designing and implementing ETL and data-ingestion pipelines.
- Understanding of incremental loads, watermarks, and merge patterns.
- Knowledge of integrating cloud data platforms with On-Premises Data Gateways.
- Experience implementing CI/CD processes and promoting data solutions across environments.
- Strong SQL skills and ability to produce clear technical documentation.
- Experience delivering work within Agile sprint teams and background in banking or financial services.