Overview
We are seeking an experienced Azure Data Engineer to join a financial services data team during a critical period of improvement following a major data platform implementation. In this role, you will collaborate with the existing team to take ownership of data engineering and pipeline development, while also contributing to the overall architecture and mentoring other engineers. Your expertise will be crucial in refining the data strategies and practices while sharing knowledge and challenging the team's existing approaches.
Responsibilities
- Build and enhance data pipelines using Databricks and Azure technologies.
- Participate in the development of a Medallion Architecture for improved data management.
- Implement Infrastructure as Code using Terraform to streamline workflows.
- Contribute to data engineering best practices and architecture improvements.
- Mentor junior engineers and aid in their professional development.
- Facilitate incremental data processing and change data capture (CDC) strategies.
- Utilize GitHub and GitHub Actions for version control and CI/CD practices.
- Engage in team discussions to challenge existing approaches and contribute innovative ideas.
Requirements
- Proven experience as a Data Engineer with a strong background in data engineering principles.
- Hands-on expertise with Databricks and Azure data services.
- Experience building data pipelines leveraging Medallion Architecture.
- Familiarity with temporal and bi-temporal data concepts.
- Strong proficiency with GitHub, GitHub Actions, and Terraform for infrastructure automation.
- Demonstrated ability in implementing CDC and incremental data processing methodologies.
- Excellent communication skills, with a capability to mentor and provide constructive feedback to peers.
- Experience in the financial services sector is advantageous but not mandatory.