Overview
The Data Engineer will be an integral part of a large-scale team focused on modernizing data infrastructure and enhancing data engineering capabilities within a global technology organization. The role involves building and maintaining robust data ingestion pipelines in a cloud environment, collaborating with various data sources, and implementing efficient data processing strategies.
Responsibilities
- Build and maintain data ingestion pipelines using cloud-based data platforms.
- Connect to multiple data sources including SQL databases, APIs, and cloud services.
- Develop automated pipelines using orchestration tools and lakeflow technologies.
- Implement appropriate ingestion strategies including incremental and batch loads.
- Develop ETL/ELT processes using modern medallion architecture frameworks.
Requirements
- Proven experience in data ingestion and engineering within a cloud environment.
- Strong proficiency in Python for data engineering and pipeline development.
- Experience with data transformation tools such as dbt.
- Familiarity with Azure Data Factory and SQL integration.
- Ability to implement scalable and maintainable data architectures.