Overview
The Senior Data Engineer will play a crucial role in a significant financial-services data transformation initiative by building and migrating enterprise data pipelines. The contractor will work primarily with Microsoft Fabric, Azure Data Factory, and ADLS, collaborating with Agile delivery teams to ensure seamless integration and efficient data management throughout the project lifecycle.
Responsibilities
- Build and enhance metadata-driven ingestion frameworks.
- Develop incremental loads, watermarks, and merge processes.
- Migrate ETL workloads to Microsoft Fabric Pipelines.
- Integrate cloud platforms with on-premises environments.
- Develop reusable deployment patterns and technical documentation.
- Support continuous integration and continuous deployment (CI/CD) across multiple environments.
- Test data pipelines, resolve defects, and support user acceptance testing (UAT).
- Collaborate within Agile delivery teams to achieve project goals.
Requirements
- Strong experience with Microsoft Fabric and Fabric Pipelines.
- Extensive knowledge of Azure Data Factory and ADLS.
- Proven experience in enterprise ETL and data ingestion techniques.
- Familiarity with metadata-driven frameworks and incremental loading.
- Proficient in SQL and Python programming languages.
- Experience with CI/CD and managing multi-environment deployments.
- Skilled in data mapping and pipeline testing.
- Experience within Agile environments, preferably in banking or financial services.