Overview
As a Lead Data Engineer, you will play a pivotal role in a cloud transformation and data modernization program, driving the development and optimization of enterprise-scale data solutions. You will collaborate with architects, stakeholders, and cross-functional teams to ensure the delivery of secure and scalable cloud-based data platforms, while mentoring a team of engineers. Your technical leadership will be essential in establishing engineering best practices and enhancing the overall data infrastructure.
Responsibilities
- Lead the design and delivery of enterprise-scale data engineering solutions within Google Cloud Platform (GCP).
- Develop and maintain scalable ETL/ELT pipelines, data models, and integration frameworks.
- Design and manage Infrastructure as Code (IaC) solutions using Terraform.
- Build and support automated deployment processes through GitLab CI/CD pipelines.
- Design, optimise, and maintain BigQuery datasets, schemas, and analytical workloads.
- Develop and manage Cloud Storage, Cloud SQL, and Cloud Composer solutions.
- Implement real-time data integration and Change Data Capture (CDC) solutions using Striim.
- Monitor platform performance, reliability, and operational efficiency across cloud environments.
Requirements
- Proven experience as a Data Engineer, with a focus on cloud technologies, preferably Google Cloud Platform (GCP).
- Strong proficiency in ETL/ELT processes, data modeling, and integration frameworks.
- Experience with Terraform for Infrastructure as Code (IaC) implementations.
- Proficient in CI/CD practices and tools, particularly GitLab.
- Knowledge of BigQuery, Cloud Storage, Cloud SQL, and Cloud Composer.
- Strong programming skills in Python for developing ETL applications.
- Background in implementing real-time data integration and Change Data Capture (CDC) solutions.
- Experience in data quality, governance, and compliance standards.