Overview
The Databricks Data Engineer will focus on data ingestion and engineering within a modern Azure and Databricks environment, playing a crucial role in connecting diverse data sources. Collaborating with a global organization, the engineer will build and maintain robust data pipelines to ensure efficient data processing and transformation.
Responsibilities
- Build and maintain data ingestion pipelines using Databricks.
- Connect to multiple data sources including SQL databases, APIs, and Azure services.
- Develop pipelines using Azure Data Factory and Databricks Lakeflow.
- Implement appropriate ingestion strategies, including incremental and batch loads.
- Develop ETL/ELT processes using Databricks Medallion Architecture.
- Write and maintain Python data engineering code.
- Use dbt for data transformation and modeling.
- Ensure pipelines are scalable, reliable, and maintainable.
Requirements
- Strong commercial experience with Databricks.
- Proven expertise in data ingestion processes.
- Experience with Azure Data Factory.
- Proficient in Databricks Lakeflow.
- Familiarity with SQL and API integration.
- Understanding of Medallion Architecture.
- Proficient in Python programming.
- Experience using dbt for data transformation.