Overview
As a Lead Data Engineer, you will play a pivotal role in a major cloud transformation and data modernization program, focusing on delivering enterprise-scale data solutions within a complex technology landscape. You will provide technical leadership and mentor a team of engineers while collaborating closely with architects, stakeholders, and engineering teams to ensure high-quality, secure, and scalable data solutions are achieved to meet business objectives.
Responsibilities
- Lead the design and delivery of enterprise-scale data engineering solutions within Google Cloud Platform (GCP).
- Develop and maintain scalable ETL/ELT pipelines, data models, and integration frameworks.
- Design and manage Infrastructure as Code (IaC) solutions using Terraform.
- Build and support automated deployment processes through GitLab CI/CD pipelines.
- Design, optimise, and maintain BigQuery datasets, schemas, and analytical workloads.
- Develop and manage Cloud Storage, Cloud SQL, and Cloud Composer solutions.
- Implement real-time data integration and Change Data Capture (CDC) solutions using Striim.
- Monitor platform performance, reliability, and operational efficiency across cloud environments.
Requirements
- Proven experience in data engineering within cloud environments, specifically Google Cloud Platform (GCP).
- Strong proficiency in developing ETL/ELT processes and data pipelines.
- Experience with Infrastructure as Code (IaC) using Terraform.
- Familiarity with CI/CD practices and tools, particularly GitLab.
- Solid understanding of BigQuery and cloud-based data storage solutions.
- Proficiency in programming with Python for data applications and automation.
- Experience with real-time data integration and CDC solutions.
- Ability to lead teams and mentor junior engineers while driving best practices.