Overview
This Data Engineer role offers a chance to work within a financial services organization undergoing a significant Databricks migration. The contractor will be responsible for constructing and maintaining data pipelines and the Databricks environment, collaborating closely with delivery teams to ensure effective project execution and support data transformation efforts. This position is primarily remote, with occasional visits to the Leeds office, and has an initial contract length of three months with potential for extension.
Responsibilities
- Build and maintain the Databricks environment and Azure data infrastructure using Terraform.
- Design, build, and own data pipelines encompassing ingestion, transformation, and orchestration.
- Monitor data quality, pipeline reliability, and performance to identify issues proactively.
- Collaborate with delivery squads on sprint planning, prioritization in Jira, and project kick-offs.
- Support data migration and integration efforts with third-party platforms such as Snowflake and Palantir.
Requirements
- Expertise in Databricks, specifically in environment build, maintenance, and optimization.
- Hands-on experience with Azure Terraform for infrastructure as code.
- Strong experience in data pipeline development, both batch and streaming.
- Proficient in SQL and Python programming skills.
- Experience with Azure DevOps and GitHub for CI/CD pipelines is a plus.