Overview
The Databricks Engineer will be an integral part of a dynamic data and AI team, tasked with designing, developing, and optimizing scalable data solutions in a fully remote role. This position focuses on enhancing a Lakehouse platform within a modern Azure and Databricks environment, collaborating closely with Data Engineers, Architects, and other technology professionals to ensure efficient and robust data pipelines.
Responsibilities
- Design and develop data pipelines and solutions using Azure Databricks.
- Optimize existing data workflows to enhance performance and scalability.
- Collaborate with Data Engineers and Architects to integrate systems and services.
- Implement CI/CD processes and modern DevOps practices.
- Support the development and management of the Lakehouse platform.
- Utilize Apache Spark and Delta Lake to streamline data processing.
- Manage data governance and security using Unity Catalog.
Requirements
- Proven experience with Azure Databricks and Azure data services.
- Strong proficiency in Python, PySpark, and SQL.
- Experience with Apache Spark and Delta Lake technologies.
- Knowledge of data pipeline and Lakehouse architecture.
- Familiarity with CI/CD and DevOps methodologies.
- Desirable: Experience with Lakeflow Connect, Lakehouse Federation, Terraform, or Azure Data Factory.